What your lab gets
Eval design
What to measure and how, turned into a set of items that actually tests it.
Item writing
Questions, tasks and reference answers written to your guidelines.
Grading
Model outputs graded against a rubric, with notes on why each grade was given.
Harnesses
Code that runs the eval, collects outputs and scores them the same way each time.
Kinds of eval work
Built to your spec and kept inside your own tools.
Capability evals
Items that test one skill, from easy to hard, with reference answers.
- items
- answers
- difficulty
Rubric grading
Outputs scored against a written rubric, with the reason for each score.
- rubric
- score
- rationale
Held-out sets
Fresh items written for a benchmark and kept out of training data.
- new items
- split
- notes
Coding evals
Programming tasks with tests, so correctness is checked automatically.
- task
- tests
- pass/fail
Domain evals
Items written and checked by experts in the field being tested.
- expert items
- review
- answers
Eval harnesses
The code that runs the set, logs outputs and reports scores.
- runner
- logs
- report
Who does the work
Engineers and domain experts, matched to what the eval measures.
- Software engineers for coding evals and for the harnesses that run every eval.
- Domain experts for items where judging an answer needs real subject knowledge.
- Graders calibrated against your rubric before and during the work.
Four checks before work counts.
The same process runs on every OceanLab Labs project, and your QA always has the final say.
- 1
Scored trial task
Every person completes a scored trial task before they join your project, so only people who meet the bar work on your data.
- 2
Second reviewer
Submitted work is checked by a second reviewer before it reaches you.
- 3
Lead calibration
A lead runs calibration rounds so everyone applies your guidelines the same way as the work goes on.
- 4
Your QA decides
Your own QA has the final say on what is accepted. Anything you reject comes back to us and is fixed.
Your data stays yours.
NDA for every person
Everyone who works on your project signs an NDA before they see any of your material.
Your tools, your accounts
Work happens only inside your own tools and accounts. Your data does not leave them.
A dedicated team
The people on your work do not work for a competing lab at the same time.
Start small, then scale.
- 1
Tell us what you need
Share the work, the skills it needs, your guidelines or rubric, and your timeline. We match people to those requirements.
- 2
Paid pilot, small team
A small team works on your real tasks first, so you can judge the output before you commit to more.
- 3
Scale up
When the pilot meets your bar, we add people to the same work and keep it running for as long as you need.
Pricing is scoped per project, usually after a short paid pilot. See how it works.
Can you keep benchmark items out of training data?
The work happens only inside your own tools and accounts, and the team works to your handling rules for held-out items.
Do you build the harness or only the items?
Both. Engineers can build the code that runs the eval and scores the outputs, alongside the items themselves.
How do you check quality?
Each person passes a scored trial task before joining, a second reviewer checks submitted work, a lead runs calibration rounds, and your own QA has the final say. Anything you reject is fixed.
How is the work priced?
Pricing is scoped per project, usually after a short paid pilot on your real tasks. There is no public rate card.
How do you keep our data and tasks private?
Everyone on your project signs an NDA, the work happens only inside your own tools and accounts, and the team does not work for a competing lab at the same time.
Related pages.
- OceanLab Labs overview Everything OceanLab Labs does for AI labs, on one page.
- Coding dataCoding tasks, reference solutions, tests and reviews of model-written code.
- RL environmentsEnvironments, tasks and checkers for training agents with reinforcement learning.
- Human feedbackRLHF and SFT data: rating, ranking, writing and correcting model outputs.
- Domain expertsSpecialists who write and check tasks where subject knowledge decides the answer.
- How it worksFrom request to paid pilot to scale: quality, security, pricing and time zones.
Looking to automate your company's workflows with AI instead? See AI automation.
