Rev Accuracy Analysis

We submitted files to Rev in order to measure their true accuracy and provide empirical evidence of their product quality. 

What We Measured

Spelling and Grammatical Errors

Word Error Rate (WER)

The WER measures the number of substitutions,  deletions, and insertions, divided by the number of words in the reference.

Order & Delivery Process

We measured the time elapsed from ordering to order confirmation and time elapsed from order received to delivery.

Common Errors

Throughout files, we saw errors such as extra and incorrect words inserted into sections with no dialogue, false starts, misplaced commas, and spelling errors.

Spelling

Incorrect Word

Missing Speaker Identifications

The Support Experience

We submitted a ticket to Rev Support regarding the high number of errors of one of the files submitted.  

Read on for the results.

Revision and Support Experience

We also measured the time elapsed for editing and correcting the returned transcripts from Rev. 

Since punctuation errors are subjective, we measured accuracy for both including and excluding punctuation and case.

We submitted 8 files with good quality audio, of varied subject matters, and with a duration of 5+ minutes.

We submitted 8 files with good quality audio, of varied subject matters, and with a duration of 5+ minutes  to Rev. 

The key takeaway:

Rev does not measure their 99% accuracy using real data, nor can they point to actual metrics to validate this claim. 

Rev did not offer to reprocess the file. 

The issue was never resolved.

Rev's average accuracy rate for the 8 files submitted ranged between 84.7% to 94.4% accuracy

The industry standard for caption quality is a 99% accuracy rate. This means that there is a 1% chance of error or a leniency of 15 errors total per 1,500 words.

The FCC, DCMP, and WCAG have all set captioning quality standards to ensure captions are comprehensible. Failure to uphold these standards could affect your brand, messaging, or result in a costly lawsuit.

For more information on caption quality, read Best Practice for Caption Quality.

"Real Earth Summit"

"periodicity in 

chemical properties"

"immigrate"

"freedom of speech"

"Chief (faint

 speaking) Officer"

"German 

Lothar Meyer"

"read"

"overthrew"

"going"

"Pendragon"

"German 

Lothan Meyer"

"readen"

"overthrow"

"gonna"

"Penn Dragon"

Could you please explain how you calculated that the file was still 99% accurate? 

We don't use an exact measurement to determine the 99%.

I've gone through the entire caption file and have denoted all mistakes in order to demonstrate why I'm unsatisfied with the results. By my count, the transcript is ~90% accurate. 

In determining if a captions file meets our accuracy guarantee, we take into account whether the error is major. 

Order Details

Time from Order Made to Order Received:

Average Caption File Turnaround Per Video:

13 minutes

2 hours, 35 minutes

Rev demonstrates quick turnarounds and low-cost rates. However, based on our findings, Rev does not demonstrate, nor can they guarantee 99% accuracy.

The Bottom Line on Quality

© 3Play Media, Inc.

Rev's process and lack of QA lead to inconsistencies throughout a single file. As a result, quality is sacrificed for quicker turnarounds.

Delays in Publishing:

File's unsatisfactory accuracy rate of 85-94% required additional editing time and a futile support conversation.

Rev's Errors:

  • Incorrect words
  • Extra words
  • Missed words
  • False starts

"German Lothan Meyer"

"readen"

"overthrow"

"gonna"

"Penn Dragon"

"German Lothar Meyer"

"read"

"overthrew"

"going"

"Pendragon"

Correct Words

Rev's Errors

Rev's Errors:

  • Use of inexistent words
  • Incorrect spelling of names and titles
  • Incorrect tense usage

Why spelling matters:

  • Affects your branding 
  • Affects the meaning of the content
  • Spelling errors distract the reader

Rev's Error:

  • Missing speaker identifications 

Why speaker IDs matter:

SHLOMO

MAITAL:

The-

The-



The-

The-

"Rio Earth Summit"

"periodicity and 

chemical properties"

"emigrate"

"free speech"

"chief product

 officer"

Correct Words

Correct Transcript

Rev's Transcript

  • Helps distinguish speaker
  • Important when there are multiple speakers
  • Incorrect wording creates confusion for the reader
  • Impacts reading comprehension

Rev's transcription process largely contributes to the inconsistencies and errors found within each file. 

Why Are There Errors?

Each individual file that comes through is cut up into smaller segments. 

These segments are then distributed across a pool of transcribers.

Once the transcription is complete, Rev pieces back together the original file.

Note: There is no quality assurance process. The Final file is never reviewed before being sent back to the customer. 

As a result of Rev's process the customer must spend more time and money reviewing and fixing the inconsistencies of the final file.

Why this matters:

Why this matters:

Rev's Errors:

"Rio Earth Summit"

"periodicity and  chemical properties"

"emigrate"

"free speech"

"chief product officer"

"Real Earth Summit"

"periodicity in chemical properties"

"immigrate"

"freedom of speech"

"Chief (faint speaking) Officer"

Correct Words

Rev's Errors

"German Lothan Meyer"

"readen"

"overthrow"

"gonna"

"Penn Dragon"

"German Lothar Meyer"

"read"

"overthrew"

"going"

"Pendragon"

Correct Transcript

Rev's Transcript



The-

The-

SHLOMO

MAITAL:

The-

The-

Punctuation

Rev's Errors:

  • Missing commas
  • Missing apostrophes
  • Missing hyphens
  • Incorrect capitalization

Why punctuation matters:

  • Affects meaning of the content
  • Affects reading comprehension

"German-speaking"

"of its 400,000"

"a state-dominant"

"architecture, I was"

"German speaking"

"of it's 400,000"

"a state dominant"

"architecture! I was"

Correct Words

Rev's Errors

GET STARTED WITH 3PLAY MEDIA

Don't Let Inaccurate Captions Hurt Your Content

3Play Media guarantees a 99% accuracy rate for every file.

(In fact, we have a measured accuracy rate of 99.6%!)

Videos Accuracy Rate Including Punctuation Errors Accuracy Rate Omitting Punctuation Errors*
Video 1
Duration: 16 min, 32 sec
90.06% 98.47%
Video 2
Duration: 16 min, 18 sec
83.63% 94.78%
Video 3
Duration: 9 min, 12 sec
94.42% 98.43%
Video 4
Duration: 14 min, 32 sec
89.8% 97.67%
Video 5
Duration: 9 min, 36 sec
83% 92.82%
Video 6
Duration: 17 min, 34 sec
81.8% 91.06%
Video 7
Duration: 5 min, 4 sec
78.23% 95.92%
Video 8
Duration: 8 min, 59 sec
76.86% 85.66%

Here's a comparison of the files we submitted alongside their accuracy rate.

Rev's average accuracy rate including punctuation for the 8 files submitted: 84.7%

Even when we omit punctuation, Rev's accuracy falls short of 99%, and the average accuracy rate for the 8 files comes to:  94.4%

*Why include or omit punctuation? Punctuation errors can be subjective, so we measured both cases. 

At 3Play Media, our accuracy rate is measured including punctuation