How Much Does It Cost to Review and Correct Caption Files In-House?
In our 2017 State of Captioning survey, 95% of respondents said they review their caption files for accuracy before publishing at least some of the time. But how much time does it take to review and correct caption files, and what is the cost of that time?
Based on internal data and our own experience with editing and reviewing caption files over the last 9 years, we’ve come up with an estimated cost of in-house QA (quality assurance) for captions.
What Factors Make QAing Captions More Time Consuming?
Depending on the original accuracy rate of your caption file, the time and cost involved in cleaning up errors can vary widely. There’s a big difference between making small tweaks and making significant edits. Regardless, you have to allot at least 1x the duration of the video just to review the captions. Any edits will add time on top of that.
The main factors that dictate QA difficulty include:
- Poor audio quality
- Complex or niche content
- Speakers with accents
- Fast speakers
- Overlapping speakers
When any of the above are true, the more time consuming it will be to correct a caption file. The key here is that errors in your caption file are most likely to occur because one (or more) of the above factors is true. This means that the person QAing your captions has to listen, pause, rewind, listen again, and correct the error(s). In really difficult cases, they may have to listen to the same section many times over, increase the volume, watch the speakers on screen, and research terms or names they’re not familiar with.
While reviewing for textual errors, the QA process should also include a review of caption timings. Changing the time codes of your caption files is time consuming because it needs to be precise. This requires moving the playback bar incrementally to get the exact timing for each caption frame.
It’s also important to be on the lookout for words that sound right but are actually incorrect, like there/their/they’re or your/you’re. Similarly, differentiating terms like “can” and “can’t” transform the meaning of the file and are very easy to miss if the QA worker is not paying close attention.
How Much Time Does It Take to QA Captions?
The time it takes will vary based on the original accuracy rate of your caption file. Let’s look at the time and cost involved in QAing a 99% accurate caption file, a 95% accurate caption file, and a caption file generated by automatic speech recognition (we’ll refer to these as automatic captions).
Cost will vary depending on who is reviewing and correcting files. If student workers are doing most of the work, we can assume an hourly rate of $15 per hour. However, if staff members are QAing your caption files, the hourly rate will be much higher. Because of these possibilities, let’s assume a range of $15-$40 per hour. Similarly, student workers may only work 20 hours per week, but full time employees work 40 hours. We’ll look at this range, as well.
Make sure to use your own personnel costs instead if the numbers below vary from your internal numbers:
Student Employee Pay $15/hr Up to $40/hr Time Works 20 hours a week Works 40 hours a week
Reviewing and Correcting a 99% Accurate File
Even at 99% accuracy, on average one in every 100 words will have an error. Files with 99% accuracy shouldn’t require extensive editing, but will require tweaking here and there. Errors may often occur on difficult terms or proper nouns. The time spent pausing, rewinding, playing, researching accurate spellings of proper nouns/difficult terms, and correcting these minor errors can add up.
We’d estimate that QAing a 99% accurate file would take about 1.3x real time, which is a reasonable task to manage in-house. For a one hour video, this QA job would cost $19.50-$50 and take about 6.5% of a student worker’s weekly time or 3.3% of a full-time employee’s weekly time.
Reviewing and Correcting a 95% Accurate File
For a typical sentence length of 8 words, a 95% word accuracy rate means there will be an error on average every 2.5 sentences. Files that are 95% accuracy can require extensive editing, particularly because error rates that high often occur when speech is difficult to understand. A QA worker will spend a lot of time rewinding, re-listening, slowing the playback down, watching the speakers, and turning the volume up.
We’d estimate that QAing a 95% accurate file would take about 4x real time. For a one hour video, this QA job would cost $60-$160 and would take up 20% of a student worker’s weekly time or 10% of a full-time employee’s weekly time.
Reviewing and Correcting Automatic Captions
Under ideal conditions, automatic captions can be as good as 80% accurate, but can be as bad as 50% accurate in cases of poor audio quality, complex content, speaker accents, and overlapping speech. Realistically, they’ll fall somewhere between 50-80%. At accuracy rates this low, QAing automatic captions becomes just as labor-intensive as transcribing from scratch (and can sometimes take even more time).
We’d estimate that QAing an automatic caption file would take about 6x real time. For a one hour video, this QA job would cost $90-$240 and would take up 30% of a student worker’s weekly time or 15% of a full-time employee’s weekly time.
In addition to the time to review and manually edit caption files, there are other editing costs to keep in mind, like:
- Making sure captions are grammatically correct
- Making sure captions are accurately synced and complete
- Making sure captions include non-speech elements
- Making sure captions include speaker identification
- Making sure captions are consistent
Fixing these errors can be even more time-consuming and costly.
The chart below shows that the lower the accuracy rate, the more time and money it will cost you to review and correct errors in-house:
Compromising on quality from the get-go can lead to expensive internal costs. If your original captions are consistently below 99% accuracy, these costs have to be considered in your budget.
At scale, these costs can multiply quickly and can make consistently delivering high-quality captions difficult.
Is internal QA taking you too long? 3Play Media guarantees 99% accuracy on all files, regardless of content type.