« Return to video

Australian Research Study: How Useful Are Closed Captions for Learning? [TRANSCRIPT]

PATRICK LOFTUS: All right. Thank you all for joining this webinar today, entitled Australian Research Study– How Useful are Closed Captions for Learning? I’m Patrick Loftus from 3Play Media, and I’ll be moderating today. And I’m very excited today to be joined by Dr. Chris Tisdell, who is the first author on the closed captioning study we’re here to discuss today.

He’s also the associate dean of science and professor of mathematics at the University of New South Wales in Sydney, Australia. Chris also has a YouTube channel, which he featured his university-level math videos on, and has reached over 10 million viewers from all around the world. And with that, I’ll go ahead and hand it off to Chris, who has a great presentation prepared for you all.

CHRIS TISDELL: OK. Thanks, Patrick. Hi, everyone. Thanks for tuning in today. I know for some of you, it’ll be the end of the day. So even after a hard day’s work, you come in and tune into one of these webinars. So thank you very much for coming in and showing some interest. Secondly, thanks to 3Play Media, Patrick, and Lily for giving me the opportunity to speak with you today and to share my results.

So as Patrick said, I’m Chris. And I’m speaking to you from the University of New South Wales in Sydney, Australia. Now the research that I’m going to share with you today is joint work with my co-author Birgit Loch.

So I wanted everyone to just sit back, relax, and imagine a world where sound and language are no barriers to learning. Just imagine that, how amazing would that be, where sound and language don’t really come into the equation. There are no barriers for learning.

Now let me tell you a little bit about my university and where the research for this university was conducted. UNSW stands for the University of New South Wales. Essentially, it’s a large state university in Sydney. We have about 55,000 students. And at the moment, we’re going through a big strategic change called UNSW 2025. And there are three pillars, if you like, for this change.

The first is academic excellence, the second is social engagement, and the third is global impact. So within that social engagement pillar, we’re looking to be a just society, and contribute as a university to transform lives, and really enable a just society. And the two pictures here– the picture on the right is of our new vice chancellor, Ian Jacobs. And the picture on the left is Merlin Crossley our deputy vice chancellor of education.

And as part of the strategy, the following quote’s quite powerful. “Our success will have been built upon embracing the diversity and cultural richness of our communities and ensuring that our staff and students can achieve their full potential.” So that really struck a chord with me when I read it because I’m really interested, as an educator, not only in scale and online learning, but also in equity, diversity, and inclusion. So that’s how this particular research study aligns with where the university is headed towards 2025.

So I’m sure most of you know this, but this is a slide that I felt that I have to include. I know there’s been a lot of press over this class action lawsuit over the MOOCs from MIT and Harvard last year regarding the failure to provide closed captioning for these online lectures in the MOOCs and various courses and podcasts and other materials.

And the following quote from the NAD, representatives of the NAD is very, very powerful. So “just as buildings without ramps bar people who use wheelchairs, online content without captions excludes individuals who are deaf or hard of hearing.” So to me, that’s sort of a compliance, legal avenue that really is just so important regarding closed captioning.

So as you can probably tell from the audio, I speak with an Australian accent. And one thing I noticed when I first started doing YouTube is that YouTube has this auto closed captioning feature. So you upload a video, and YouTube has this automatic speech-to-text type algorithm. Now one thing I’ve noticed is that the algorithm doesn’t like a diverse range of accents.

So, for example, I speak with an Australian accent. YouTube doesn’t really know what to do with an Australian accent. And I would say, any kind of accent that is a little step away from some of the American accents– so for example, Scottish, Irish, English, South African– these sort of different accents are not captured well with the auto closed captioning on YouTube.

So here’s a couple of screenshots. It’s just from one of the videos. My background is in mathematics, and the study that I’m looking at today is about an introductory mathematics course. You can see even on the top left-hand screenshot, the words are actually typed on the page. Hence, factorize p of z into linear factors. And then you look down at the closed captions, and it says, “and heads fact arise he upset.” Now, down on the bottom right-hand corner, we’ve manually put in the correct closed captions there. So that’s an example of where the automatic closed captioning really falls over.

Now part of this is accent, like I said. So it doesn’t really know what to do with an accent like mine. But it’s also partly scientific terms as well. So, for example, in mathematics, and science, and perhaps engineering, there are a lot of technical scientific terms. I think that is a struggle, too, for the auto closed captioning.

So there’s a problem. What do we do about it? Well, I’m glad you asked. So I got a team of students together. They were all volunteers, about 10. And we spent a summer together manually creating closed captions for around 45 of my educational videos. So these videos would be taken by a university student who was doing some introductory, basic math course. And so the audience was engineering students, science students, business students, and anyone who needs some basic first year of mathematics.

All right. So we didn’t have any training in the creation of closed captioning, but one thing that we really knew there was a need for was to get standard notation. So everybody was working with the same notation. Now that ensured consistency.

I got the students to work in teams to create the closed captions. So basically, they were paired up, and they were given maybe two or three videos to caption, each pair. They would swap and check each other’s work. And then they would give me the final draft, and I would check it and then upload it.

Now we didn’t give any particular prescription for the method of creating the closed captioning. I’m sure a lot of it was quite inefficient. So, for example, one pair of students told me that they were typing on their computer screen while watching the video on an iPad next to the computer screen. You kind of need two screens. So I’m sure that it was quite inefficient. But what I wanted to do in that sense was to just get an idea of what the diversity in the methods were being used.

Now once we had the final transcript and we uploaded it to YouTube, YouTube was quite good at auto-syncing the text or the captioning with the videos. So that was quite fine.

So onto the experiment or the next stage of the experiment. Like I said, this was an introductory math course, essentially on complex numbers. We encouraged our first-year students to watch these YouTube videos though they weren’t prescribed as part of any kind of class. They were just an additional learning resource. So if the students wanted some out-of-class help, they could go and watch these videos. And we monitored the usage over a two-month period.

Now just one limitation of the research is the videos had the closed captioning turned on as your default setting. So if I’m a student and I watch a video, the closed captioning would come up automatically. And they could turn it off if they wanted to. All right?

And after the two-month period, we surveyed the students. So the picture here is not the real class, but it’s a typical class where I teach. So it’s a little bit like “Where’s Wally?” You get five points if you can see me. All right.

So on to the results now. The blue bars are the students’ responses to the question, closed captioning was useful for my learning. And it’s a classic six-point Likert scale. And you can see we split up the cohort into different groups, so domestic students– they’re Australian or New Zealand students; international students; native English speakers; ESL, English as a Second Language; deaf students. And then we’ve combined everything together in the final blue column on the right.

So you can see, just looking at the blue columns, they’re pretty good scores. They’re all above five. Some of them are 5.5. You can see the deaf students. Only two deaf students, though, I’ve got to say. There’s a limitation there.

There were around 135 students responding to the survey. And you can see international is– the blue column is slightly above domestic. ESL students felt a little bit strongly than native speakers of English. But still, native speakers of English saw benefits in it.

Just to explain the scale, one is strongly disagree, two is disagree, three is mildly disagree, four is mildly agree, five is agree, and six is strongly agree. So we asked the students, overall, closed captions were useful to my learning. And then they either agreed, or disagreed, or whatever. So it was pretty good. I think overall, it was about 5.35 out of 6.

Now you’re probably looking and saying, well, Chris, what’s the orange columns? Well, again, I’m glad you asked. The orange columns are about the automatic translations available on YouTube. So let me talk about that just for a minute. YouTube has auto translate that goes to about 90 different languages. And we were getting that for free once we’d already uploaded the manual closed caption transcript.

So we thought we would just ask. And you can see the reactions are not quite as strong as the blue closed captioning columns, but still pretty good. You can see the international students are about 5. So overall, on average, they would agree that translations were useful to their learning– not as strong, though.

So here I’ve just put it into percentage agree. And this really is a very, very strong indicator that the students felt that closed captioning was very, very useful. You can see domestic students, above 95%, international students, 100%. So every international student either mildly agreed, agreed, or strongly agreed that closed captioning was useful for their learning. That’s an amazing result. Even native English speakers still saw lots of value in closed captioning. You can see ESL, English as a Second Language, students, again, 100%.

So overall– and this is one of the take-home messages, I guess, from this. Overall, 98% of students in this study broadly agreed that closed captioning was useful for their learning. 98%. And I know that’s on par with some of the other studies that have been done.

Now one of the interesting things we thought about when we started doing this project was, OK, well, this will be great for international students. It’ll be great for students who have English as a second or third language. We’re not so sure if the closed captioning will be that useful for native English speakers.

Well, we were completely wrong about that. We got some messages and comments back from people who had comments like this. “I speak and understand English just fine, but I like closed captionings because I can sit on the train without headphones and watch your videos.” “If I’m in a noisy place, I can watch your videos.” So that really surprised us, because we were thinking, oh, yes, this will be great for people who are hard of hearing, people who need help with their English. Actually, it was warmly welcomed by everyone.

So what did we learn? Like I said, closed captioning is not just useful for people who are hard of hearing or have language difficulties. The results of this survey show that closed captioning is useful for everyone. I’m sure some of you are not that familiar with an Australian accent. So even though you speak English as a first language, it’s still helpful sometimes to have closed captioning as an option. So the educational benefits of closed captioning go way beyond the usual expected audiences and situations. I think that’s really important.

This is my last slide, so the change that we want to see. Well, from an Australian perspective, we don’t really have any national guidelines for closed captioning for online video. We have closed captions guidelines for television broadcasts and the like, but that’s something that I think is important for online video because, let’s be honest, online video is just going to grow, and grow, and grow. And secondly, just happy to have further discussion around the best practice and the scale in closed captioning for online video.

So that’s my last slide. Patrick, shall I pass back to you?

PATRICK LOFTUS: Sure. I will take over now, Chris. Thank you so much. All right. Well, again, Chris, thank you so much, first of all, for joining us so early in the day and for giving such an interesting, insightful presentation. So we’re going to get started on Q&A in a moment. Let’s get started on Q&A.

First question here, Chris. You mentioned issues with accents. Did international and ESL students respond that they found captions most useful for understanding accents, or was it more correlated to helping them better comprehend the English content?

CHRIS TISDELL: I think it’s a combination of both. One thing– I mean, I’ve been in higher education for about 15 years. And in Australia, we have a lot of international students. And one of the common complaints here is that the lecturer, the tutor, they speak with a strong accent. English might not be the first language of their teacher or of their tutor. So accents is really important. And being able to manage accents and understand a thick accent is really important.

Now there were some students who identified that they’ve found benefits. One of the benefits in using closed captioning was you’re kind of accent-independent, which I found was great. Even on YouTube, on my videos on YouTube sort of just around the world, I get people saying, oh, nice accent, or funny accent, or I can’t understand your accent. So I think it’s a big issue. And we’re a very diverse world. So we all speak a little bit differently.

PATRICK LOFTUS: I agree. Yes. Thank you, Chris. And the second question here, someone’s asking, are you planning to conduct this study again with a control group and focus on if it improved their learning?

CHRIS TISDELL: Yeah, that’s a really good point. So that is another limitation of the research. We would love to do that. Part of having the control group is a little bit tricky because we need to make sure that we are fair in the sense that we’re not excluding people from gaining access to certain resources. But that is something that I would love to pursue maybe in the next phase of the project. And that’s certainly a deeper, and more serious, and more important element.

So at the moment, what we’re doing– we’re just essentially looking at perceptions. Now some people think that student perceptions don’t hold much weight. I would argue against that. Our university, we get the best students in the state with the highest entry scores. We teach them to be critical thinkers. I think that listening to the student voice is important. But certainly, I would love to have a bit more of a controlled approach and be able to take the next step.

PATRICK LOFTUS: Thanks, Chris. Next question here, do you have best practices for captioning math content and equations?

CHRIS TISDELL: Not yet. That is one discussion point that I would love to unpack a bit more. Mathematics is tricky because it has difficult or sometimes complicated expressions. It’s the same with physics, or chemistry, or any of the technical and scientific fields. So when you’re working with a basic ASCII type keyboard, it’s very tricky.

So we’ve tried to be able to replicate equations where we can. It’s not like you’re captioning a speech by Obama, where it’s just basic words. It’s very tricky.

One of the things you’ll notice about if you go and watch some of my videos is the equations are kind of already on the screen. So if a viewer can already see an equation on the screen, is it necessary to actually have the captioning again? That’s one of the questions that we’ve faced. I don’t have an answer to that.

But certainly if you’re going to take this to the more scientific fields with closed captioning, the technical equations and whatnot, they’re going to be a barrier. And that’s something we need to explore, whether it’s

LayTeX integration with closed captioning, which is a beautiful typesetting program. Maybe that’s the future.

PATRICK LOFTUS: Great. Thanks, Chris. Next question here– were there any instances where students strongly disagreed that captioning was useful to their learning?

CHRIS TISDELL: Let me have a look. I just got my paper in front of me now. Let’s have a look. Certainly for the translations, there were some strongly disagrees. There were maybe 1% of students strongly, strongly disagreed. And about 1% mildly disagreed. And there were no disagrees.

PATRICK LOFTUS: Interesting. The next question was actually, did anyone note that the translations were not helpful because of their quality?

CHRIS TISDELL: Yes.

PATRICK LOFTUS: Do you have any more info on that?

CHRIS TISDELL: Yes. That’s a really good question, too. So we were relying on Google Translate pretty much. I think that’s embedded in the YouTube translation service. That was a big bear bug from some of the international students or ESL students.

The quality of the translations is not quite there yet. So maybe in five years or wherever, it’ll be much better. But some of the students said that, OK, some of the translations are wrong, but we can still get an idea of what’s going on. So it’s not 100% accurate, but you can get a general idea of what’s going on.

PATRICK LOFTUS: Great. Thanks, Chris. Next question here is, can you confirm how uploading transcripts to YouTube works? How does it auto-time? Does the transcript require specific formatting for this to work properly?

CHRIS TISDELL: The sky’s the limit here. You can get special files, SRT files, or something like that with all time-stamped text and whatnot. We took the simplest approach possible. So what we did was we got the students to work with a Word file, just a simple Word file. And then they would send the Word file to me, and I would match it up with the video. And basically, you’re just copying the text out of the Word file and putting it into a little box when you want to edit the closed captioning for a particular video on YouTube. It was fine.

The syncing was fine. Automatically, it was very, very easy to do. So you can have really amazing programs like– I think it’s VTT or SRT files, something like that. But we didn’t go that approach. We went the simplest approach that we could all do, which was using text in a Word file that we copied and paste into the appropriate YouTube video place for each video. It was very easy. And the alignment between the captioning and the words, YouTube did a very good job of doing that. And it was automatic.

PATRICK LOFTUS: Great. Yeah, thank you for the walk-through, Chris. And the next one here is, how do you think– this is pretty interesting. How do you think research will help generate buy-in for captioning at universities that currently only caption for accommodation requests?

CHRIS TISDELL: Yeah. So my university is the same. You have to ask for these sorts of things if you want them. I think, as a society, universities should be making evidence-based decisions and having good research– not just from this study, but I know you guys have been doing stuff in the US as well– that’s evidence-based and evidence-informed decision-making.

So I think that’s really important when we can say that this is something– first of all, this is something that legally we should be providing. Secondly, the students want this. There is a huge market for this. The students see a lot of benefit in this. That’s a pretty power– And the evidence is that the students highly value this. That’s a pretty compelling argument. So I think now we’re in a position to really move this forward. And that’s what I’m hoping to do.

PATRICK LOFTUS: And leading to the next question here, what areas do you see for future research?

CHRIS TISDELL: I think that the translations is still a bit of a– sort of a horizon event here in the sense that how can we improve the experience when using translations? How can we improve the accuracy of translations? That’s really important. I think we still haven’t solved the captioning of technical parts, like equations, scientific formulae, that sort of thing. I think that’s important. I know you guys at 3Play Media are doing a lot with the interactive transcripts. I see that as sort of really going amazing places. So there are two or three ideas that just come to mind straightaway.

PATRICK LOFTUS: Great. Thank you, Chris. And I guess we’ll do one more because we’re running out of time here. Last question is, do you think that these results would be the same for other subject matters? Or do you think that they were so positive for math because of its complexity?

CHRIS TISDELL: Mm. That’s a good question. One of the challenges with social science research is the ability to replicate experiments. It’s a bit tricky.

[SIGH]

Yeah, I don’t know. This is sort of discipline-based in math, and science, and engineering, STEM, I guess. It would be interesting to look at the research that’s been done outside of STEM. So if somebody’s thinking about doing that, I’d be very happy to help. I would hope it would be the same. I’m not sure why it wouldn’t. But that remains to be seen.

PATRICK LOFTUS: Great way to leave it. Thank you, Chris, so much for that great presentation and for sticking around for Q&A. And thank you to everyone for joining us. Keep an eye out tomorrow for an email with a link to view the recording and slide deck. And have a great rest of your day, everyone.

CHRIS TISDELL: Great. Thanks, everyone. Bye.