What a Rating Task Looks Like

Response B fails on safety and correctness at once. The NHS says not to give aspirin to children under 16 because of a rare risk of Reye's syndrome, and it advises seeing a doctor if a child's fever lasts 5 days or more, not two weeks. A follows the NHS advice, so A is much better.

Sources: NHS: fever in children, NHS: aspirin.

How the AI Labs Describe a Good Answer

OpenAI published the instructions its human raters followed when training InstructGPT, in the appendix of its 2022 paper. Raters judged three things:

For final evaluations, truthful and harmless usually came first. The one exception: prefer the more helpful answer only if it is much more helpful, barely worse on safety, and the topic is not high stakes like medical, legal or money advice. When still unsure, they were told to ask which answer they would rather get from a customer assistant.

The same paper shows the priority was flipped during an earlier training phase. That is the most useful lesson for new raters: general advice is a default, and your project's written instructions always win.

Where Your Ratings Go

The InstructGPT paper lays out the three training steps that rating and writing work feeds:

1. People write ideal responses to prompts, and the model is fine-tuned to imitate them.
2. People rank several model responses from best to worst, and those rankings train a reward model that predicts which answer people prefer.
3. The model is then trained with reinforcement learning to produce answers the reward model scores highly.

That is why careless ratings matter. Each ranking becomes part of what the model is taught to prefer.

Sources: OpenAI InstructGPT paper.

Check in This Order

Rules Good Raters Follow

1. Read the project guideline fully before the first task, then re-read its rules on severity and ties.
2. Verify facts yourself instead of trusting confident wording.
3. Do not reward length. A shorter correct answer beats a longer padded one.
4. Check every explicit constraint in the prompt: word count, format, language, persona.
5. Use a tie only when the guideline allows it and the responses are genuinely equal.

Sources: OpenAI InstructGPT paper, Anthropic 2022 study, r/DataAnnotationTech thread.

What Google Tells Its Raters

Google's Search Quality Rater Guidelines (September 2025 edition) are written for search results, but three ideas carry straight over to AI responses:

• Top scores are rare. The guidelines say most queries “cannot have a Fully Meets result”. Over-rating is the classic beginner error.
• Trust matters most. Of Experience, Expertise, Authoritativeness and Trust, trust is the most important.
• Some topics need stricter checks. Health, money, safety and civic topics can seriously affect people, so even a small error counts heavily. The fever example above is one of these.

Two Traps the Research Found

• Polish is not truth. In Anthropic's 2022 study, raters often preferred answers containing links that did not work. Check the facts and the links, not the formatting.
• A refusal is not automatically the safest good answer. The same research notes that an answer explaining why a request is harmful is better than a bare refusal.

Disagreement is also normal. OpenAI's raters agreed with each other about 73% of the time, and Anthropic measured about 63% agreement between researchers and crowdworkers. Calibration with reviewers is part of the job, not a sign you are failing.

Your ratings are checked in layers. At our top pick, peer reviewers check work for accuracy and guideline adherence, an operations team may check it again, and some projects add a client review.

How Much Better: Using the Scale

Project guidelines define these levels in their own words. When they do, use theirs.

Common Rating Mistakes

• Picking the longer or friendlier answer without checking facts.
• Missing a constraint, such as a word limit, that one response ignored.
• Letting a nicely formatted answer hide a wrong fact or a broken link.
• Rating from memory of the guideline instead of checking it.

Video and Audio Annotation

Some AI training work has no text to rate at all. You watch or listen, then mark where something happens: when an action starts and ends, what kind of event it is, or what someone said and when they said it.

Mercor's public job board shows how this work is described. On 2026-10-09 it listed eight Audiobook QA Expert roles across seven languages, where reviewers log each narration error and mark the exact span where it occurs. In August 2026 it listed a first-person video project where reviewers checked whether each action segment's start and end matched the action. Ethos‘s expert terms also name annotation among the project work experts take on.

Sources: Mercor job board, Ethos expert terms.

What the Tasks Look Like

• Segment labeling means marking the start and end of a moment or action, then tagging it with a label from the project's list.
• Boundary review means checking segments that someone else or a model marked, then adjusting or flagging the ones that are off.
• Transcription work asks you to write down what is said, fix speech-to-text errors and tie each line to the moment it is spoken.
• Audio review asks you to log problems such as skipped, added or mispronounced words, numbers read wrongly and audio glitches, put each in a category and mark where it happens.

How to Be Precise

• Work in frames when the tool allows it. At 30 frames per second one frame lasts about 33 milliseconds, and at 25 frames per second it lasts 40. A Mercor video editing listing asks for frame-accurate cuts, and the BBC's subtitle guidelines write times in frames too.
• Check every boundary at the frame level. Find the moment at normal speed, then step one frame at a time either side of your mark before you save it.
• Write a default rule for unclear starts and ends, with a unit, such as “start on the first frame of contact, end on the first frame the hand lets go”. Use the project's own rule wherever it has one.
• Keep one unit per project. Frame 300 is 10 seconds at 30 frames per second but 12 seconds at 25, so note each clip's frame rate and convert to seconds or milliseconds when the tool asks for time.
• Time speech from its first sound. The BBC guidelines say a subtitle should appear as speech starts, and a new speaker's line as that person starts to speak.

Expect boundaries to be hard. In a 2017 study, crowd workers who re-marked actions in two video datasets overlapped the original labels by about 72.5% and 58.7%. Ends were harder to place than starts, with median errors of about 1.4 seconds against 0.9, and short actions were the hardest of all, so give them extra care.

Sources: Sigurdsson et al. 2017, BBC subtitle guidelines, Mercor job board.

Staying Consistent Over Long Sessions

The Mercor video review listing asked for “consistent judgment across a high volume of short action segments”. These habits help:

• Keep an edge-case log. Each time you make a call the guideline does not cover, write the rule down with one example timestamp, so the next case gets the same call.
• Re-check a sample. Reopen a few of your earliest segments after a batch and compare them with your latest ones; drift shows up there first.
• Calibrate with reviewers. Read their feedback, and ask the project lead in the project channel when your rule and theirs differ. Trainers on Reddit say guidelines change often and that the project chat is the place to ask.
• Take breaks. Trainers on Reddit warn that very long sessions lead to autopilot and slipping quality, and an audiobook listing asks for accuracy across many hours of audio.

Sources: Reddit thread, Reddit thread, Reddit thread.

Common Annotation Mistakes

• Ending segments too late or too early. The end of an action is usually less clear than its start, so check ends twice.
• Setting boundaries at normal playback speed without stepping through frames.
• Mixing frames and seconds, or reusing frame numbers between clips with different frame rates.
• Labeling what you think happened instead of what you hear. The BBC guidelines say a sound label should describe the sound itself, such as floorboards creaking, rather than the action behind it.
• Changing your own rule halfway through a batch without going back to fix the earlier segments.