Cielo24 Accuracy Analysis

We submitted files to Cielo24 in order to measure their true accuracy and provide empirical evidence of their product quality. 

What We Measured

Spelling and Grammatical Errors

Word Error Rate (WER)

The WER measures the number of substitutions,  deletions, and insertions, divided by the number of words in the reference.

Common Errors

Throughout files, we saw errors such as extra and incorrect words inserted into sections with no dialogue, false starts, misplaced commas, and spelling errors.

Punctuation

Spelling

Incorrect Word

Read on for the results.

Since punctuation errors are subjective, we measured accuracy for both including and excluding punctuation and case.

We submitted 8 files with good quality audio, of varied subject matters, and with a duration of 5+ minutes.

We submitted 7 files with good quality audio, of varied subject matters, and with a duration of 5+ minutes to Cielo24. 

Cielo24's average accuracy rate for the 7 files submitted ranged between 84.85% to 91.97% accuracy

The industry standard for caption quality is a 99% accuracy rate. This means that there is a 1% chance of error or a leniency of 15 errors total per 1,500 words.

The FCC, DCMP, and WCAG have all set captioning quality standards to ensure captions are comprehensible. Failure to uphold these standards could affect your brand, messaging, or result in a costly lawsuit.

For more information on caption quality, read Best Practice for Caption Quality.

 Cielo24 does not guarantee 99% accuracy.

The Bottom Line on Quality

© 3Play Media, Inc.

Cielo24's process and lack of QA lead to inconsistencies throughout a single file. As a result, quality is sacrificed.

  • Incorrect use of periods
  • Incorrect use of commas
  • Incorrect use of apostrophes
  • Incorrect capitalization

Cielo24's Errors:

Why punctuation matters:

"LHC of CERN and its experiments"

"So during this course I will"

"For one thing, obviously, there's"

"LHC of CERN, and it's experiments"

"So during this course. I will"

"for on thing, obviously there's"

Correct Words

Cielo24's Errors

"LHC of CERN and 

its experiments"

"So during this 

course I will"

"For one thing, 

obviously, there's"

"LHC of CERN, and 

it's experiments"

"So during this 

course. I will"

"for on thing, 

obviously there's"

  • Extra words
  • False starts
  • Incorrect word use

"centre"

"the food taste different"


"center"

"the food tastes different"

Correct Words

Cielo24's Errors

"center"

"the food tastes 

different"

"centre"

"the food taste 

different"

Cielo24's Errors:

  • Improper tense use
  • Incorrect spelling

Why spelling matters:

  • Affects your branding 
  • Affects the meaning of the content
  • Poor spelling distracts the reader
  • Incorrect wording creates confusion for the reader
  • Impacts reading comprehension
  • Affects meaning of the content
  • Affects reading comprehension
  • Incorrect punctuation confuses and distracts the reader

Cielo24's transcription process largely attributes to the inconsistencies and errors found within each file. 

Why Are There Errors?

Each individual file that comes through is cut up into smaller segments. 

These segments are then distributed across a poll of transcribers.

Once the transcription is complete, Cielo24 pieces back together the original file.

Note: There is no quality assurance process. The Final file is never reviewed before being sent back to the customer. 

As a result of Cielo24's process the customer must spend more time and money reviewing and fixing the inconsistencies of the final file.

Why proper word use matters:

Why this matters:

Rev's Errors:

"Play-in is the reverse"

"For one thing,"

"building in general should"

"Play as a reverse"

"One, for one thing,"

"building in general, should be should"

Correct Words

Cielo24's Errors

"Play as a reverse"

"One, for one thing,"

"building in general,

 should be should"

Cielo24's Errors:

"German Lothan Meyer"

"readen"

"overthrow"

"gonna"

"Penn Dragon"

"German Lothar Meyer"

"read"

"overthrew"

"going"

"Pendragon"

Correct Transcript

Rev's Transcript

"Play-in is the reverse"

"For one thing,"

"building in general

 should"

Cielo24's Errors:

3Play Media guarantees a 99% accuracy rate for every file.

(In fact, we have a measured accuracy rate of 99.6%!)

Don't Let Inaccurate Captions Hurt Your Content

GET STARTED WITH 3PLAY MEDIA
Videos Accuracy Rate Including Punctuation Accuracy Rate Omitting Punctuation*
Video 1
Duration: 6 min, 46 sec
85.87% 93.94%
Video 2
Duration: 8 min, 18 sec
85.64% 92.72%
Video 3
Duration: 1 min, 52 sec
82.51% 88.21%
Video 4
Duration: 13 min, 4 sec
82.2% 92.49%
Video 5
Duration: 16 min, 27 sec
82.49% 90.7%
Video 6
Duration: 2 min, 5 sec
86.34% 90.12%
Video 7
Duration: 14 min, 18 sec
88.88% 95.62%

Here's a comparison of the files we submitted alongside their accuracy rate.

*Why include or omit punctuation? Punctuation errors can be subjective, so we measured both cases. 

At 3Play Media, our accuracy rate is measured including punctuation 

Cielo24's average accuracy rate for the 7 files submitted: 84.85%

Even when we omit punctuation, Cielo24's accuracy falls short of 99%, and the average accuracy rate for the 7 files comes to: 91.97%