PART 2
Task Description: In this part of the SPL trial, you will be responsible for overseeing a critical piece of every project’s success: performance management. Owning this process ensures that we maintain a positive relationship with our clients by prioritizing delivery of high-quality work within an allotted time frame. Note this is self contained from part 1.
Loom video about part 2 (10 minutes)
Campaign Description: Project Elephant is a research project currently being carried out by one of our clients (a leading AI lab in the United States). The purpose of this project is to create an evaluation set that assesses the LLM’s ability to perform complex online research tasks at an “expert” level across domains such as law, engineering, and linguistics. As such, experts are asked to create a series of really difficult “prompts” (or research questions) in their domain and then construct a detailed rubric outlining the characteristics of a good answer such that researchers can understand how well the LLM is performing.
Talent Description: The talent that Mercor has sourced for this project consists of highly-educated (typically have masters degrees or PhDs) individuals from top institutions in the US in law, engineering, linguistics, business, and journalism. The hourly pay rates are as follows:
Engineering: $90/hr
Law: $90/hr
Linguistics: $50/hr
Business (includes Consulting, Marketing, Finance, etc.): $90/hr
Generalist (includes journalists): $70/hr
Goal: The goal of the project is to create as many approved (“signed off”) high-quality prompt-rubric pairs (called “tasks”) as possible. In order to ensure high quality, the pipeline can be split into two phases: writing and reviewing. Our experts can either be classified as “trainers” (writing tasks only) or “reviewers” (writing and reviewing others’ tasks). Trainers have a limit of 20 hours per week and reviewers have a limit of 40 hours per week.
A few key things to note:
The average task takes 6 hours to write, 2 hours to review per cycle, and 1.5 hours to revise per cycle.
We expect the average task to undergo 2 revision cycles before sign-off. However, this may differ based on the quality of the task writer (a strong writer may only require 1 revision cycle, for example).
Data
This spreadsheet contains all other data provided for this part of the onsite.
Pipeline Description: After a task is “claimed”, a trainer will work on it until they designate it as “Complete.” At that point, the task will be sent to the review pile, where a reviewer will examine it for minor and/or major issues. The reviewer has two options:
1) put the task in “Signed Off” if it is (nearly) perfect or
2) mark the task as “Needs Work” if it requires additional revisions
If the task is marked as “Signed Off”, that means it is an approved task and counts towards our goal! If “Needs Work”, then it gets sent to the original task writer, who will make the necessary revisions and then put it in “Fixing Done.” At that point, a reviewer will pick up the task again and the process repeats until the task is finally “Signed Off”.
Questions to Answer:
Our task completion target within the next 2 weeks is 250 “Signed Off” tasks total. Are we on track to hit this goal?
If not, what course of action would you suggest for reaching this target with high quality? You can suggest as many initiatives/changes as needed, but be detailed in your explanation.
Some things to consider:
We should continue to performance manage the experts during this period to make sure we maintain high quality.
Ten members of the review team are “team leads”, which means they have extensive experience in this project.
We have a sourcing pipeline that adds experts each week, but the sourcing team needs to be given sourcing targets with 4 day lead times from request to onboarding.
We do have an additional $7,500 available in the weekly budget, on top of the base rates we’re paying.
You can change anything in the process except for three things:
No expert can work more than 40 hours per week
The hourly base pay rates are fixed
Every task must be reviewed at least once before sign-off