LLM Evaluation Specialist
cypress hcmMillbrae, CA
$59.86/hour
APPLY (opens in a new tab)LLM Evaluation Specialist
cypress hcmMillbrae, CASenior L5
$59.86/hour
Other Scientific and Technical Consulting ServicesOther Management Consulting ServicesAll Other Professional, Scientific, and Technical Services
Location: Hybrid San Francisco, CA 94107 (Open to Remote)
Duration: 11/16/2026 - 12/18/2026
Schedule: 15 hours per week
Team:
Incubator - VizWhiz
Job Details LLM Evaluation Specialist
Location: Hybrid San Francisco, CA 94107 (Open to Remote)
Duration: 11/16/2026 - 12/18/2026
Schedule: 15 hours per week
Team: Incubator - VizWhiz
About the Job:
Work with a small team building an AI-native data storytelling product that helps business professionals turn data into clear, honest narratives.
Run recurring evaluation passes on AI-generated charts, headlines, captions, and chat responses against a defined rubric.
Judge accuracy, chart-type fit, clarity, and tone; flag misleading framing, specious correlations, and awkward phrasing.
Log findings in evaluation tooling and deliver a concise report to the team after each cycle.
Suggest rubric improvements based on observed patterns.
Part-time role: approximately 15 hours per week over a multi-week contract.
Duties:
Perform recurring evaluation passes on AI-generated charts, headlines, captions, and chat responses using the team’s rubric.
Assess whether claims are supported by the underlying data and verify chart-type fit.
Identify and flag misleading framing, specious correlations, and unclear or awkward phrasing not caught by automated evaluations.
Record findings in evaluation tooling and produce concise post-cycle reports for the team.
Recommend concrete rubric changes informed by recurring error patterns.
Required Qualifications:
Strong data analysis and data visualization judgment.
Sharp editorial instincts and high attention to detail.
Comfortable working with spreadsheets and evaluation tools (no coding required).
Clear, concise written reporting skills.
Bachelor’s degree in a related field such as data analytics, statistics, journalism, information science, or business, or equivalent practical experience.
Preferred qualifications: Prior experience evaluating LLM output.
Compensation Pay Rate: $59.86/hour
Matching similar jobs
Level
Senior L5
Salary
$59.86/hour
Location
Millbrae, CA
Occupation
Data Scientists
Industry
Other Scientific and Technical Consulting Services
Posted
3 days ago