Senior Research Engineer - Evaluations

About the Role

Key Responsibilities

• Design, build, and optimize the infrastructure for an "MLLM-as-a-Judge" evaluation system for scalable, automated feedback.
• Implement and experiment with inference-time alignment techniques (Prompt Engineering, RAG, ICL) to directly improve model output quality.
• Establish and manage a comprehensive benchmarking process to compare various foundation models on design-centric tasks.
• Analyze evaluation data to identify model failure modes and provide actionable recommendations to the research team.
• Collaborate with research scientists and ML engineers to integrate the agentic judge system into the model development lifecycle.
• Translate the latest research in LLM evaluation and agentic AI into practical, production-ready engineering solutions.

Requirements

• Strong understanding of generative AI models (e.g., Diffusion Models, GANs, Transformers) and their architectures, with practical experience that informs robust evaluation strategies.
• Experience creating data-driven evaluation methodologies, turning user analytics into clear, actionable insights.
• Experience managing or optimizing large-scale distributed model training across hundreds of GPUs.
• Solid understanding of machine learning, with experience working with PyTorch and optimizing code for speed.
• Disciplined coding practices, and experience with code reviews and pull requests.
• Experience working in cloud environments, ideally AWS.

Nice to Have

• Familiarity with evaluation libraries and frameworks.
• Experience building or working with agentic AI systems or multi-agent coordination.
• Knowledge of data visualization tools to communicate findings effectively.
• A background or interest in human-computer interaction, design principles.

Qualifications

• Experience with generative AI models and evaluation strategies.
• Practical experience with large-scale distributed training and cloud environments.
• Proficiency in PyTorch and optimizing machine learning code.

Benefits & Perks

• Equity packages - we want our success to be yours too.
• Health benefits plans to support you and your wellbeing.
• 401(k) retirement plan with company contribution.
• Inclusive parental leave policy that supports all parents & carers.
• An annual Vibe & Thrive allowance to support your wellbeing, social connection, office setup & more.
• Flexible leave options that empower you to be a force for good, take time to recharge and support you personally.