CaptionSync by AST Accuracy Analysis

We submitted files to CaptionSync by AST in order to measure their true accuracy and provide empirical evidence of their product quality. 

What We Measured

Spelling and Grammatical Errors

Word Error Rate (WER)

The WER measures the number of substitutions,  deletions, and insertions, divided by the number of words in the reference.

Order & Delivery Process

We measured the time elapsed from ordering to order confirmation and time elapsed from order received to delivery.

Common Errors

Throughout files, we saw errors such as extra and incorrect words inserted into sections with no dialogue, false starts, misplaced commas, and spelling errors.

The most common error we saw from CaptionSync was [inaudible] tags when the audio was clear enough to transcribe.

Missed Words

Incorrect Words

Missing Speaker IDs

Read on for the results.

Revision and Support Experience

We also measured the time elapsed for editing and correcting the returned transcripts from CaptionSync. 

Since punctuation errors are subjective, we measured accuracy for both including and excluding punctuation and case.

We submitted 8 files with good quality audio, of varied subject matters, and with a duration of 5+ minutes.

We submitted 7 files with good quality audio, of varied subject matters, and with a duration of 2+ minutes  to CaptionSync. 

CaptionSync's average accuracy rate for the 5 files submitted ranged between 93.49% to 95.00% accuracy

The industry standard for caption quality is a 99% accuracy rate. This means that there is a 1% chance of error or a leniency of 15 errors total per 1,500 words.

The FCC, DCMP, and WCAG have all set captioning quality standards to ensure captions are comprehensible. Failure to uphold these standards could affect your brand, messaging, or result in a costly lawsuit.

For more information on caption quality, read Best Practice for Caption Quality.

"we must first study

"This ideal point"

"the bearing of the point"

"From your separate position"

"present you the major models"

CaptionSync demonstrates quick turnarounds and low-cost rates. However, based on our findings, CaptionSync does not demonstrate - nor can they guarantee - 99% accuracy.

The Bottom Line on Quality

© 3Play Media, Inc.

On average, CaptionSync only produces 93-95% accuracy.

CaptionSync's Errors:

  • Incorrect words
  • Extra words
  • False starts
  • Incorrect tense

"The rising sun moves 

<

<

so minimal 

[inaudible]."


"using the practical

[inaudible] cosmic rays"


"The center of [inaudible]

of our planet"

"The rising sun moves

from a minimal azimuth,

to the north of east,

to a maximal azimuth,

to the south of east."


"using particles 

like cosmic rays"


"The center of mass 

of our planet"

Correct Words

CaptionSync's Errors

"we must first start"

"This idea of point"

"the burning the point"

"From the observer position"

"present using major models"

Correct Words

CaptionSync's Errors:

  • Misses words
  • Uses [inaudible] tags unnecessarily

Why spelling matters:

  • Affects your branding 
  • Affects the meaning of the content
  • Affects viewer comprehension

CaptionSync's Error:

  • Missing speaker identifications 

Why speaker IDs matter:

GIULIO 

MAGLI:

Today,

we

explore



Today,

we

have [inaudible]

"The rising sun moves

from a minimal azimuth,

to the north of east,

to a maximal azimuth,

to the south of east."


"using particles

like cosmic rays"

"The rising sun moves

<                                           

<

so minimal

[inaudible]."


"using the practical

[inaudible] cosmic rays"

Correct Words

Correct Transcript

CaptionSync's Error

CaptionSync's Errors

  • Helps distinguish speaker
  • Important when there are multiple speakers
  • Incorrect wording creates confusion for the reader
  • Impacts reading comprehension

Why this matters:

Why this matters:

CaptionSync's Errors:

"we must first study"

"This ideal point"

"the bearing of the point"

"From your separate position"

"present you the major models"

"we must first start"

"This idea of point"

"the burning the point"

"From the observer position"

"present using major models"

Correct Words

CaptionSync's Errors

"German Lothan Meyer"

"readen"

"overthrow"

"gonna"

"Penn Dragon"

"German Lothar Meyer"

"read"

"overthrew"

"going"

"Pendragon"

Correct Transcript

CaptionSync's Error



Today,

we

have [inaudible]

GIULIO

MAGLI:

Today,

we

explore

Punctuation

CaptionSync's Errors:

  • Missing commas
  • Missing apostrophes
  • Missing hyphens

Why punctuation matters:

  • Affects meaning of the content
  • Affects reading comprehension

"the Earth's axis"

"azimuth-altitude coordinate"

"If you want, as we do,"

"the Earth axis"

"azimuth altitude coordinate"

"If we want as we do"

Correct Words

CaptionSync's Errors

GET STARTED WITH 3PLAY MEDIA

Don't Let Inaccurate Captions Hurt Your Content

3Play Media guarantees a 99% accuracy rate for every file.

(In fact, we have a measured accuracy rate of 99.6%!)

Videos Accuracy Rate Including Punctuation Errors Accuracy Rate Omitting Punctuation Errors*
Video 1
Duration: 2 min, 9 sec
94.39% 87.79%
Video 2
Duration: 14 min, 18 sec
98.24% 97.03%
Video 3
Duration: 16 min, 27 sec
93.90% 93.52%
Video 4
Duration: 13 min, 4 sec
97.00% 94.60%
Video 5
Duration: 1 min, 52 sec
95.87% 88.21%
Video 6
Duration: 8 min, 18 sec
94.88% 94.48%
Video 7
Duration: 6 min, 46 sec
86.75% 86.18%

Here's a comparison of the accuracy rates for the files we submitted. There was a high level of variance in accuracy rates across 7 files.

CaptionSync's average accuracy rate including punctuation for the 7 files submitted: 93.5%

Even when we omit punctuation, CaptionSync's accuracy falls short of 99%, and the average accuracy rate for the 7 files comes to:  95.0%

*Why include or omit punctuation? Punctuation errors can be subjective, so we measured both cases. 

At 3Play Media, our accuracy rate is measured including punctuation