Skip to main content

LLM Evaluation Specialist

cypress hcmMillbrae, CASenior L5

$59.86/hour

Location: Hybrid San Francisco, CA 94107 (Open to Remote)
Duration: 11/16/2026 - 12/18/2026
Schedule: 15 hours per week
Team: Incubator - VizWhiz Job Details LLM Evaluation Specialist
Location: Hybrid San Francisco, CA 94107 (Open to Remote)
Duration: 11/16/2026 - 12/18/2026
Schedule: 15 hours per week
Team: Incubator - VizWhiz
About the Job: Work with a small team building an AI-native data storytelling product that helps business professionals turn data into clear, honest narratives. Run recurring evaluation passes on AI-generated charts, headlines, captions, and chat responses against a defined rubric. Judge accuracy, chart-type fit, clarity, and tone; flag misleading framing, specious correlations, and awkward phrasing. Log findings in evaluation tooling and deliver a concise report to the team after each cycle. Suggest rubric improvements based on observed patterns. Part-time role: approximately 15 hours per week over a multi-week contract.
Duties: Perform recurring evaluation passes on AI-generated charts, headlines, captions, and chat responses using the team’s rubric. Assess whether claims are supported by the underlying data and verify chart-type fit. Identify and flag misleading framing, specious correlations, and unclear or awkward phrasing not caught by automated evaluations. Record findings in evaluation tooling and produce concise post-cycle reports for the team. Recommend concrete rubric changes informed by recurring error patterns.
Required Qualifications: Strong data analysis and data visualization judgment. Sharp editorial instincts and high attention to detail. Comfortable working with spreadsheets and evaluation tools (no coding required). Clear, concise written reporting skills. Bachelor’s degree in a related field such as data analytics, statistics, journalism, information science, or business, or equivalent practical experience.
Preferred qualifications: Prior experience evaluating LLM output.
Compensation Pay Rate: $59.86/hour
APPLY NOW (opens in a new tab)

Matching similar jobs

Level

Senior L5

Salary

$59.86/hour

Location

Millbrae, CA

Occupation

Data Scientists

Industry

Other Scientific and Technical Consulting Services

Posted

3 days ago

APPLY NOW (opens in a new tab)