Skip to main content

Evaluation Engineer

reinforced learningTonawanda, NYLead L6
The role:
  • Research designs evaluations; you make them run reliably. You'll build the harnesses, graders and reporting that let a model be evaluated on hundreds of tasks across our environments, with results anyone can reproduce.
  • What you'll own
  • The evaluation harness: task loading, environment orchestration, model adapters, scoring.
  • Graders and verifiers implemented with the environment and verifier engineers.
  • Runs that are deterministic, resumable and cheap to repeat.
  • Dashboards and reports that researchers and customers read.
  • Discrepancies between runs, chased until they're explained.
  • What you bring
  • Strong software engineering in Python with an eye for reliability.
  • Model APIs, batch systems and asynchronous processing.
  • You care about reproducibility and know how to achieve it.
  • Comfort reading research code and making it production-grade.
  • Helpful, not required
  • Open evaluation frameworks.
  • Containers and cloud infrastructure.
APPLY NOW (opens in a new tab)

Matching similar jobs

Level

Lead L6

Location

Tonawanda, NY

Occupation

Software Developers

Industry

Software Publishers

Posted

3 days ago

APPLY NOW (opens in a new tab)