What Is 99% Accuracy, Really? Why Caption Quality Matters

June 6, 2018 BY SOFIA ENAMORADO
Updated: August 6, 2018

99% accuracy is the industry standard for caption quality.

Most captioning vendors guarantee 99% accuracy.

But, how true are their claims?

What does accuracy include?

A 99% accuracy rate means that there is a 1% chance of error or a leniency of 15 errors total per 1,500 words.

Accuracy measures punctuation, spelling, and grammar.

Note: Punctuation errors can be subjective. At 3Play, we measure our accuracy including punctuation because we believe incorrect punctuation can make a tremendous difference to the comprehension of a file. In addition to including punctuation, we also have internal standards to meet linguistic and legal requirements.

Why is it 99% accuracy and not 100%?

Neither technology nor humans can deliver a 100% accuracy rate.

Automatic speech recognition (ASR) is good, but not good enough to remove humans from the process.

can you decipher youtube asr? take the quiz

ASR doesn’t include speaker identifications or important sound effects. ASR transcripts are often riddled with inconsistencies in spelling and in grammar.

That’s why it’s important to have human editors review the transcript.

Challenging our competitor’s claims

We decided to challenge 2 major competitors on their 99% accuracy claims.

We submitted several files with:

  • Good quality audio
  • Varied subject matter
  • Duration of 5+ minutes

We measured each file’s accuracy based on spelling errors, grammar errors, and word error rate.

We also measured accuracy including and excluding punctuation.

Numbers don’t shouldn’t lie

Both competitors we challenged advertise a 99% accuracy rate on all their files.

But in reality, their measured accuracy rate falls between 84.7% and 94.4%.

graph demonstrating accuracy rate. vendor accuracy rate ranges from 84.7% to 94.4%. industry standard is 99%, and 3play media measured accuracy rate is 99.6%"

At 3Play Media, we guarantee a 99% accuracy rate for every file processed (In fact, we have a measured accuracy rate of 99.6%).

What errors did our competitors make?

We uncovered numerous spelling errors and inconsistencies throughout every single file we submitted.

The most common errors made by our competitors were spelling, punctuation, incorrect wording, and false starts.

We uncovered numerous spelling errors and inconsistencies throughout every single file we submitted. Here's a taste of what we saw: Competitors Transcript Examples "Then, my favor poet E.E. Cummings came along" versus correct "Then, my favorite poet E.E. Cummings came along" competitor "Bob Nice is a co-inventor of integrated circuits" and correct ""Bob Noyce is a co-inventor of integrated circuits." competitor ...expand the coup of influence in more militant ways. and correct ...expand the scope of influence in more militant ways.

 

Grammar and punctuation are critical for reading comprehension and for compliance with captioning best practices.

Even when we removed grammar and punctuation errors from our calculations, the best accuracy rates we measured were only 92-94.4%.

Such low accuracy rates are unacceptable from an industry and accessibility perspective.

Why did our competitors make these errors?

Process directly impacts file accuracy.

Both of the competitors we tested follow a similar process.

a file is split into multiple documents that are sent to multiple editors and finally pieced back together

First, they cut a single file into smaller segments; then they distribute those segments across a pool of transcribers.

Once the transcription is complete, the file is pieced back together.

As a result, your files come back full of inconsistencies and errors.

Furthermore, our competitors also miss a critical step in the quality assurance process.

Both competitors have little-to-no quality assurance process. This means the final file is never reviewed before being sent back to the customer.

This results in inconsistencies in spelling and grammar across a single file.

For example, “mom” could be spelled “mum,” or “flyer” could be spelled “flier.” These subtle changes mean everything to the comprehension of a file.

How can 3Play gurantee 99% accuracy?

At 3Play Media, we guarantee at least a 99% accuracy rate on all your files.

Like our competitors, our process directly impacts the accuracy of our files.

We use a 3-step process that combines artificial intelligence and human editing to maximize efficiency and optimize our process.
 

AUTOMATIC SPEECH RECOGNITION  Your video will go through automatic speech recognition to produce a rough transcript. HUMAN EDITING CLEANUP  A US-based editor will efficiently clean up the rough transcript using our proprietary software. HUMAN QUALITY REVIEW  A quality assurance manager will conduct a final review of the transcript and captions.

 

Our model is designed to make the captioning process easier for you.

Every file is first put through ASR to produce a rough skeleton for our transcriptionist to edit – this step simply helps to speed up the editing process so that we can offer faster turnaround options.

Note: Your file is never cut up.

Next, one of our thousands of US-based editors takes the rough ASR transcript and begins editing it.

Finally, every file is put through a round of human quality review. A quality assurance manager conducts a thorough review of the final transcript and caption file making sure grammar and spelling are consistent and accurate.

We keep your files intact and also check them twice.

What is the cost of inaccurate captions?

99% accuracy at rates approaching $1 per minute seems too good to be true – because it is.

When we took into consideration spelling, punctuation, incorrect words, and insertions, our competitors’ accuracy rates ranged from only 84.7%-94.4% accuracy.

Incorrect grammar and punctuation make captions extremely difficult to follow – when we allowed for those errors, average accuracy was still only 92-94.4% accurate.

Inaccurate captions mean more work for you.

Why should you have to go back and edit your files, if you paid for someone else to professionally caption them?

Captioning your videos isn’t cheap.

When you pay for captioning, your vendor should be thorough and accurate.

Inaccurate captions take a lot more work on your end. Instead of focusing on other projects, your or your employees will have to spend additional time editing returned files or spend even more money reprocessing the files.

Inaccurate captions lead to miscomprehension of content.

Misspellings and incorrect words can be detrimental to the comprehension of your content.

This is especially problematic with educational videos. Mistakes in your caption file can easily misinform a student.

Inaccurate captions also hurt your brand.

A UK study uncovered that what damages a customer’s view of a brand is “poor spelling or grammar.”

Grammar errors deter the credibility of a brand.

In a study by Global Lingo, 59% of respondents said they “would not use a company that had obvious grammatical or spelling mistakes on its website or marketing material.”

Ouch.

Don’t let inaccurate captions hurt you.

How to alleviate accuracy concerns

man looking worried

To help alleviate accuracy concerns, look for vendors who:

  • Measure their accuracy rate – and can tell you how.
  • Don’t crowdsource or use offshore transcriptionists.
  • Train transcriptionists on the standards.
  • Have a method for handling difficult content.
  • Have an easy way to edit captions without having to reprocess.
  • And always make sure to ask vendors about their captioning process.

Want more tips? Download the free white paper ‘How to Select the Right Captioning Vendor: 10 Crucial Questions to Ask.


Want true 99% accuracy? At 3Play Media, we are confident in our quality and process. We guarantee a 99% accuracy for all your captions and a 100% customer satisfaction. click to give 3play media a try!

3play media logo in blue

Subscribe to the Blog Digest

Sign up to receive our blog digest and other information on this topic. You can unsubscribe anytime.



By subscribing you agree to our privacy policy.