Senior Data Engineer - Data Platform & Databricks Expert

About the Role

Key Responsibilities

• Serve as Databricks Subject Matter Expert (SME): Lead the adoption, optimization, and evangelization of Databricks and its ecosystem (Delta Lake, Unity Catalog, MLflow, Databricks SQL).
• Architect, build, and optimize big data solutions using Databricks and other modern data platforms such as Azure SQL, Synapse, and Snowflake.
• Drive our data strategy, ensuring alignment with business goals and best practices. Lead platform governance and ensure compliance with industry standards.
• Continuously optimize Databricks clusters, jobs, and workflows for performance, reliability, and cost-efficiency. Apply similar optimization principles across adjacent data platforms as needed.
• Lead the development and implementation of best practices for coding, data management, and platform operations. Mentor and upskill engineers, data scientists, and analysts on Databricks features and data engineering best practices.
• Stay updated on the latest industry trends and innovations in data engineering. Engage in Databricks forums, conferences, and user groups to share knowledge and learn from the community.
• Lead incident response efforts, troubleshoot complex issues, and implement long-term solutions.

Requirements

• At least 7 years of experience in data engineering, with a minimum of 3 years in architecting and operating Databricks solutions in production environments.
• Expert-level proficiency in Databricks, including Delta Lake and Unity Catalog.
• Experience implementing and optimizing ETL/ELT on modern platforms such as Azure SQL, Synapse, Databricks, and Snowflake, with demonstrated ability to integrate multiple data sources (including structured and unstructured data, APIs, and streams).
• Expertise in Delta Lake and Apache Iceberg formats for scalable batch and near real-time processing.
• Strong programming skills in Python and PySpark with experience developing production-grade data processing applications and optimizing Spark jobs for performance.
• Strong proficiency in SQL and modern data modeling methods (including medallion architecture, dimensional modeling, and star/snowflake schemas).
• Ability to design efficient data structures for analytics.
• Preferred: Experience building CI/CD pipelines for data platforms using Databricks CLI, Terraform, and Git-based workflows.
• Preferred: Practical experience with machine learning workflows and MLOps practices, such as model training, versioning, and deployment using Databricks ML tools and MLflow.
• Demonstrated ability to lead teams and mentor junior engineers and data scientists.
• Experience in leading migrations from legacy platforms to modern data platforms, including Databricks.
• Bachelor's or Master's degree in Computer Science, Engineering, or a related field (or equivalent experience).

Nice to Have

• Databricks certifications (e.g., Databricks Certified Data Engineer Professional).
• Experience with real-time streaming technologies (e.g., Structured Streaming, Kafka).
• Contributions to Databricks forums, conferences, or other industry communities.

Qualifications

• Bachelor's or Master's degree in Computer Science, Engineering, or a related field (or equivalent experience).

Benefits & Perks

• Comprehensive benefits package including medical, dental, vision, 401k, PTO/paid sick leave, and other benefits.
• Employee stock purchase plan.