Senior Data Engineer / DevOps Engineer

About the Role

Key Responsibilities

• Design, build, and maintain scalable distributed pipelines to support data ingestion and data processing using open-source frameworks
• Leverage tools like Apache Spark, Ray or Dask for high-performance parallel data processing
• Develop and optimize parallel processing frameworks capable of handling large-scale ingestion and model complexity
• Support distributed and parallelized ML model training across cloud environments and on-premise GPU clusters
• Containerize ML applications using Docker and deploy/manage them using Kubernetes
• Develop automated pipelines for model training, evaluation, deployment, and rollback, integrating CI/CD practices and MLOps tools
• Implement observability for ML systems using monitoring and logging tools to track model performance, data quality, and system health
• Work closely with Processing Algorithms & Data Science teams to integrate, optimize, and deploy algorithms to production
• Apply debugging and problem-solving skills to support and troubleshoot data-intensive applications in production, including on-call responsibilities
• Participate in collaborative software development practices, including merge request reviews and providing design feedback
• Work in a fast-paced agile environment, effectively communicate and track development activities using tools like JIRA/Confluence

Requirements

• B.S. degree in Computer Science, Electrical/Computer Engineering, or comparable experience
• 3+ years of professional software development experience using Python and experience with parallel and distributed computing frameworks such as Apache Spark, Ray, or Dask
• Experience with standard Python tools and frameworks (e.g., NumPy, Pandas, SciPy, SciKit)
• Experience developing and supporting DevOps best practices
• Hands-on experience with Docker containerization and Kubernetes for deployment and scaling
• Experience with Infrastructure as Code (IaC) tools such as Terraform or Crossplane
• Hands-on experience managing the end-to-end machine learning lifecycle using tools like MLflow
• Design and manage cloud-native data solutions using AWS services like S3 and RDS
• Experience with on-premise GPU clusters
• Experience with monitoring deployed models using tools such as Prometheus and Grafana

Nice to Have

• Familiarity with data streaming platforms like Apache Kafka or Amazon Kinesis
• Experience with modern data warehouse solutions such as Snowflake or BigQuery
• Familiarity with machine learning frameworks like PyTorch or TensorFlow
• Working knowledge of C++ for performance optimization and compute-intensive workloads

Qualifications

• Educational background in Computer Science, Electrical/Computer Engineering, or comparable experience

Benefits & Perks

• Competitive base salary plus annual performance bonus
• Work from home flexibility in a hybrid work environment
• Benefits package (unspecified in detail)