What Is 99% Accuracy, Really? Why Caption Quality Matters
Updated: October 7, 2019
99% accuracy is the industry standard for caption quality.
Most captioning vendors guarantee 99% accuracy.
But, how true are their claims?
What does accuracy include?
A 99% accuracy rate means that there is a 1% chance of error or a leniency of 15 errors total per 1,500 words.
Accuracy measures punctuation, spelling, and grammar.
Note: Punctuation errors can be subjective. At 3Play, we measure our accuracy including punctuation because we believe incorrect punctuation can make a tremendous difference to the comprehension of a file. In addition to including punctuation, we also have internal standards to meet linguistic and legal requirements.
Why is it 99% accuracy and not 100%?
Neither technology nor humans can deliver a 100% accuracy rate.
Automatic speech recognition (ASR) is good, but not good enough to remove humans from the process.
ASR doesn’t include speaker identifications or important sound effects. ASR transcripts are often riddled with inconsistencies in spelling and in grammar.
That’s why it’s important to have human editors review the transcript.
Challenging our competitor’s claims
We decided to challenge 2 major competitors on their 99% accuracy claims.
We submitted several files with:
- Good quality audio
- Varied subject matter
- Duration of 5+ minutes
We measured each file’s accuracy based on spelling errors, grammar errors, and word error rate.
We also measured accuracy including and excluding punctuation.
don’t shouldn’t lie
Both competitors we challenged advertise a 99% accuracy rate on all their files.
But in reality, their measured accuracy rate falls between 84.7% and 94.4%.
At 3Play Media, we guarantee a 99% accuracy rate for every file processed (In fact, we have a measured accuracy rate of 99.6%).
What errors did our competitors make?
We uncovered numerous spelling errors and inconsistencies throughout every single file we submitted.
The most common errors made by our competitors were spelling, punctuation, incorrect wording, and false starts.
Grammar and punctuation are critical for reading comprehension and for compliance with captioning best practices.
Even when we removed grammar and punctuation errors from our calculations, the best accuracy rates we measured were only 92-94.4%.
Such low accuracy rates are unacceptable from an industry and accessibility perspective.
Why did our competitors make these errors?
Process directly impacts file accuracy.
Both of the competitors we tested follow a similar process.
First, they cut a single file into smaller segments; then they distribute those segments across a pool of transcribers.
Once the transcription is complete, the file is pieced back together.
As a result, your files come back full of inconsistencies and errors.
Furthermore, our competitors also miss a critical step in the quality assurance process.
Both competitors have little-to-no quality assurance process. This means the final file is never reviewed before being sent back to the customer.
This results in inconsistencies in spelling and grammar across a single file.
For example, “mom” could be spelled “mum,” or “flyer” could be spelled “flier.” These subtle changes mean everything to the comprehension of a file.
Anecdotal feedback corroborates the claim that long-form content (full lectures, TV episodes, etc.) will have major consistency and quality issues.
How can 3Play guarantee 99% accuracy?
At 3Play Media, we guarantee at least a 99% accuracy rate on all your files.
Like our competitors, our process directly impacts the accuracy of our files.
We use a 3-step process that combines artificial intelligence and human editing to maximize efficiency and optimize our process.
Our model is designed to make the captioning process easier for you.
Every file is first put through ASR to produce a rough skeleton for our transcriptionist to edit – this step simply helps to speed up the editing process so that we can offer faster turnaround options.
Note: Your file is never cut up.
Next, one of our thousands of professional editors takes the rough ASR transcript and begins editing it.
Finally, every file is put through a round of human quality review. A quality assurance manager conducts a thorough review of the final transcript and caption file making sure grammar and spelling are consistent and accurate.
We keep your files intact and also check them twice.
What is the cost of inaccurate captions?
99% accuracy at rates approaching $1 per minute seems too good to be true – because it is.
When we took into consideration spelling, punctuation, incorrect words, and insertions, our competitors’ accuracy rates ranged from only 84.7%-94.4% accuracy.
For a sentence of 8 words, a 95% accuracy rate means there will be an average of 2.5 errors in every sentence.
Incorrect grammar and punctuation make captions extremely difficult to follow – when we allowed for those errors, average accuracy was still only 92-94.4% accurate.
Inaccurate captions mean more work for you.
Why should you have to go back and edit your files, if you paid for someone else to professionally caption them?
Captioning your videos isn’t cheap.
When you pay for captioning, your vendor should be thorough and accurate.
Inaccurate captions take a lot more work on your end. Instead of focusing on other projects, your or your employees will have to spend additional time editing returned files or spend even more money reprocessing the files.
Inaccurate captions lead to miscomprehension of content.
Misspellings and incorrect words can be detrimental to the comprehension of your content.
This is especially problematic with educational videos. Mistakes in your caption file can easily misinform a student.
Inaccurate captions also hurt your brand.
A UK study uncovered that what damages a customer’s view of a brand is “poor spelling or grammar.”
Grammar errors deter the credibility of a brand.
In a study by Global Lingo, 59% of respondents said they “would not use a company that had obvious grammatical or spelling mistakes on its website or marketing material.”
Don’t let inaccurate captions hurt you.
How to alleviate accuracy concerns
To help alleviate accuracy concerns, look for vendors who:
- Measure their accuracy rate – and can tell you how.
- Don’t crowdsource.
- Train transcriptionists on the standards.
- Have a method for handling difficult content.
- Have an easy way to edit captions without having to reprocess.
- And always make sure to ask vendors about their captioning process.
Want more tips? Download the free white paper ‘How to Select the Right Captioning Vendor: 10 Crucial Questions to Ask.‘
Instagram Introduces Closed Captions for IGTV
On September 15, 2020, Facebook announced a new feature for Instagram: closed captions for IGTV videos powered by AI. Yes – you heard that right! IGTV, Instagram’s platform for user-generated on-demand videos, now offers auto-captioning capabilities in 16 languages with plans to…
What INBOUND 2020 Speakers Had to Say About Accessibility
From marketers to developers, INBOUND 2020 brought hundreds of speakers, thousands of people, and millions of ideas for organizations to stand out in our virtual world. Below you’ll hear how INBOUND incorporated accessibility into their remote event, as well as the biggest…
Strategies for Transitioning to Accessible Remote Learning
3Play Media’s second virtual event, ACCESS at Home, focused on providing the necessary tools and knowledge to thrive in a virtual environment. Dawn Hunziker and Barbie Lopez, the IT accessibility team for the University of Arizona, joined us for ACCESS at Home…