Senior Data Engineer

About the Role

Key Responsibilities

• Lead regular design and code reviews to enforce development best practices and ensure high-quality, maintainable pipelines across the team.
• Monitor and optimize job and query performance in Databricks and Airflow, identifying bottlenecks and implementing scalable solutions.
• Contribute to the architecture and design of modern data platform solutions leveraging Databricks and Airflow, with a focus on performance, scalability, and cost-efficiency.
• Author and maintain clear documentation on pipeline architecture, operational workflows, data models, and platform configurations.
• Partner with leadership to drive adoption of modern platform features and implement data governance standards across the organization.
• Design and implement production-grade data pipelines for real-time and batch workloads, including infrastructure and operational monitoring.
• Mentor junior engineers by promoting platform best practices, sharing design patterns, and encouraging innovation and ownership.
• Communicate platform insights and improvements through dashboards, models, or concise reports that help stakeholders make data-driven decisions.

Requirements

• 5+ years of development experience in a core programming language, preferably Python, with strong skills in coding, debugging, and testing.
• 5+ years of hands-on experience with Databricks or an equivalent big data platform (e.g., GCP BigQuery, AWS Redshift), including building and optimizing ETL/ELT pipelines in Delta Lake or similar data lake formats.
• Familiarity with Unity Catalog, SQL Warehouses, and performance tuning in Databricks.
• 5+ years of experience using Apache Airflow (preferably with Astronomer or MWAA), including designing modular, idempotent, production-ready DAGs, and experience with dynamic DAG generation, task retries, and sensor management.
• Strong experience working in cloud environments like AWS, Azure, or GCP, including proficiency in cloud-native services such as S3, IAM, Lambda, EventBridge, and Secrets Manager.
• Understanding of cloud networking, security, and data storage best practices.
• Experience with Infrastructure as Code tools, preferably Terraform, for managing Airflow and Databricks deployments.
• Familiarity with CI/CD practices, particularly as they relate to data platform tooling (e.g., GitHub Actions, Azure DevOps, Buildkite).
• Exposure to data quality, observability, and lineage tools, such as Monte Carlo or Datafold.
• Working knowledge of dbt and its role in the modern data stack, especially when integrated with Databricks.
• Experience building REST APIs or microservices to support data access or automate platform operations.

Nice to Have

• Experience with dynamic DAG generation, task retries, and sensor management in Airflow.
• Knowledge of data quality, observability, and lineage tools like Monte Carlo or Datafold.
• Familiarity with dbt and its integration with Databricks.
• Experience with REST API or microservice development for data access or automation.

Qualifications

• 5+ years of development experience in a core programming language, preferably Python.
• Experience with Databricks or equivalent big data platforms.
• Hands-on experience with Apache Airflow (preferably with Astronomer or MWAA).
• Experience working in cloud environments such as AWS, Azure, or GCP.
• Experience with Infrastructure as Code tools like Terraform.

Benefits & Perks

• Health plans including options as low as $0 per paycheck, with programs for fertility, family planning, mental health, and fitness.
• Generous paid time off (PTO & sick leave), annual bonus, and long-term incentive opportunities based on performance.
• 401k with up to a 5% match, commuter benefits, pet insurance, and more.
• Paid personal time off and 14 paid company holidays.
• Paid sick time in accordance with applicable laws.
• Competitive salary range of $138,000 - $181,650 USD, dependent on experience and location.
• Additional benefits may include medical, vision, dental insurance, life insurance, disability insurance, and stock participation.