Machine Learning Engineer (Real-Time Speech Translation)

What you’ll do: Build and run services that supporthigh-throughput, real-timeaudio and text streaming. Own signal processing, session lifecycle management, andconcurrencyto keep the system stable under load. Integrate and servestreaming speech recognitionandmachine translationmodels, working with research teams to meetlatency budgets. Develop model-driven confidence scoring and routing logic forhuman interventionwhen needed. Broadcast real-time updates and corrections to end users. Architect and scale production ML infrastructure onGPU-accelerated Kubernetes, includingRay Servedeployments. Implement batching, load balancing, and autoscaling strategies to maintain both performance and cost efficiency. Set up instrumentation to measure real-time performance and establish production-grade observability. Identify bottlenecks, optimize end-to-end throughput, and reduceend-to-end latencyto meet production standards. Define technical contracts and interfaces for audio ingestion and downstream service integrations. Collaborate with frontend and platform engineering teams to maintain robust integration points. Drive cross-team alignment through clear API interfaces and technical contracts across engineering and product teams.

Key requirements: BS or MSin Computer Science (or related) or equivalent practical experience. 3+ yearsbuilding production backend or ML serving systems inPython, with strongasyncskills (asyncio). Hands-on experience with real-time streaming transport such asWebSocketorgRPC bidirectional streaming, including session state, backpressure, and connection lifecycle handling. Experience serving ML models on GPUs in production usingRay Serve, Triton, vLLM, or similar, along withDockerandKubernetes. Experience integrating speech or NLP models into production systems, ideally streaming ASR with partial hypotheses, endpointing, and VAD. A latency-engineering mindset: you have profiled, instrumented, and optimized real-time or low-latency systems and can reason using per-stage budgets. Ability to use AI coding agents (Claude Code, Codex, or similar) with strong fundamentals: you can debug, review, and reason about every line and know when not to trust generated output. US citizenship and residencein the United States (contract requirement). TechnologiesPython, asyncio, WebSocket, gRPC Ray Serve, Triton, vLLM Docker, Kubernetes Claude Code, Codex COMET, CometKiwi RabbitMQ Datadog, Prometheus WebRTC, SFU LiveKit Agents, Pipecat Location and eligibilityBoston, MA (onsite) RequiresUS citizenship and residencein the United States. Preferred locations include Washington, D.C.; Boston, MA; and Indianapolis, IN (East Coast / ET timezone preferred). CompensationUSD 120,000 - 161,434per yearly.

Preferred qualifications: Ray Serve experience, including streaming responses and model multiplexing. Familiarity with simultaneous or incremental MT concepts (retranslation, prefix stability, wait-k policies). Machine translation quality estimation in the COMET/CometKiwi class, or other production confidence estimation. Message brokers for real-time fan-out and state distribution (RabbitMQ or similar). Streaming text-to-speech integration and time-to-first-audio optimization. WebRTC and SFU concepts, or voice pipeline frameworks such as LiveKit Agents or Pipecat. Handling CJK and other non-Latin text in NLP pipelines (Japanese, Korean, and English are first languages). Observability tooling for production ML systems (Datadog, Prometheus).