Qualtrics logo

Senior Software Developer in Test, Foundation - Seattle

Qualtrics

Seattle, WA
Full Time
Senior
140k-255k
about 1 month ago

Job Description

About the Role

At Qualtrics, we create software the world's best brands use to deliver exceptional frontline experiences, build high-performing teams, and design products people love. We are more than a platform—we are the creators and stewards of the Experience Management category serving over 18K clients globally. Our teams are high-functioning, collaborative, and dedicated to serving our customers, working in a culture that encourages innovation, strategic risks, and growth opportunities. The company values connection, empathy, and quality in everything we do, with a focus on building reliable and responsible AI solutions.

Key Responsibilities

  • Drive the quality assurance strategy for agentic features, ensuring standardization of testing approaches from a customer-centric perspective
  • Develop comprehensive evaluation tools and frameworks that enable automated testing of AI agent workflows
  • Define model launch readiness criteria for agentic features that align with both customer needs and regulatory requirements
  • Establish model risk management protocols that document AI system capabilities and limitations
  • Create knowledge transfer processes to disseminate best practices in AI testing across Qualtrics teams
  • Design and implement automated evaluation frameworks for testing AI agent behaviors and outputs
  • Build upon existing AI test strategy and LLM-as-a-Judge infrastructure to create comprehensive testing solutions
  • Develop metrics and benchmarks to quantify AI system quality, reliability, and safety
  • Create reproducible test scenarios that simulate real-world user interactions with AI agents
  • Collaborate with product teams to integrate quality processes into their AI development workflows
  • Identify and mitigate potential risks in AI agent behaviors, including hallucination detection and security guardrails
  • Present testing results and quality metrics to stakeholders across engineering and product teams

Requirements

  • Bachelor's degree in Computer Science or related field
  • 5+ years of professional coding experience with strong Python skills
  • 2+ years experience building test automation frameworks and tools from scratch
  • 5+ years of relevant experience as part of an automated test team, with 1+ years as a senior contributor
  • Understanding of Machine Learning concepts (deep learning, categorization, classification, data mining)
  • Experience with Machine Learning evaluation frameworks
  • Experience with CI/CD deployment automation (Jenkins, etc.)
  • Proficiency with testing distributed systems and SOA (Service Oriented Architecture)
  • Experience applying various software testing techniques (equivalence class partitioning, boundary value testing, pairwise testing)
  • Strong critical thinking and root cause analysis skills

Nice to Have

  • Experience working on cutting-edge AI technologies and infrastructure
  • Knowledge of security guardrails for AI models
  • Experience with large language model (LLM) evaluation systems
  • Leadership in developing innovative testing methodologies for AI systems

Qualifications

  • Educational background in Computer Science or related field

Benefits & Perks

  • Wellness Reimbursement: $300 per quarter for wellness activities including gym memberships, spa massages, workout equipment, meditation apps, and more
  • Experience Bonus: $1800 to be used for an "Experience" of your choosing
  • Access to QGroup Communities such as MOSAIQ, Green Team, Qualtrics Pride, Q&Able, Qualtrics Salute, and Women's Leadership Development
  • Hybrid work model with in-office presence three days a week and flexible remote work for the rest

Working at Qualtrics

Our team works on cutting-edge AI technologies and is pioneering quality engineering practices for AI systems. We foster a culture of collaboration, innovation, and quality, actively working on innovative projects like security guardrails, LLM evaluation systems, and agentic workflows. We value continuous learning, technical leadership, and building infrastructure that enables reliable, safe, and effective AI features. The environment encourages strategic risks, growth, and a shared mission to bring humanity, connection, and empathy back to business.

Apply Now

Job Details

Posted AtJul 3, 2025
Job CategoryQA Engineering
Salary140k-255k
Job TypeFull Time
Work ModeHybrid
ExperienceSenior

Job Skills

AI Insights

Key skills identified from this job posting

Sign upto access all insights for this job

About Qualtrics

Website

qualtrics.com

Company Size

5001-10000 employees

Location

Seattle, WA

Industry

Surveying and Mapping (except Geophysical) Services

Get job alerts

Set up personalized alerts for your job search and get tailored job digests for close matches