Databricks Engineer

About the Role

Key Responsibilities

• Implement scalable data solutions on Databricks to drive business insights and operational efficiency.
• Translate complex business requirements into technical solutions, leveraging Databricks and modern data platforms.
• Lead strategic data migrations from legacy RDBMS systems to cloud-native Databricks environments with minimal disruption.
• Design and optimize data pipelines that support enterprise-wide analytics, reporting, and machine learning initiatives.
• Develop robust ETL frameworks using Databricks to ensure high-quality, timely, and secure data delivery.
• Continuously improve data workflows for performance, scalability, and cost-efficiency, aligning with business KPIs.
• Collaborate with cross-functional teams—including product, analytics, and IT—to align data solutions with organizational goals.
• Establish and enforce data engineering best practices, ensuring consistency, quality, and compliance across projects.
• Evaluate emerging technologies and recommend tools that enhance data capabilities and business value.
• Mentor junior engineers, fostering a culture of continuous learning and technical excellence.
• Operationalize machine learning pipelines on Databricks, from data preparation to model deployment and monitoring.
• Partner with DevOps and infrastructure teams to ensure reliable and secure deployment of data systems.
• Ensure compliance with data governance, privacy, and regulatory standards, integrating controls into data workflows.
• Align data architecture with enterprise IT strategy, working closely with architects and leadership to support long-term scalability.

Requirements

• Proven experience delivering data-driven solutions using Databricks, with a focus on business impact and scalability.
• Databricks certifications and hands-on expertise in building ETL pipelines, data lakes, and ML workflows.
• Experience with cloud platforms such as AWS, Azure, or GCP, and understanding of their data ecosystems.
• Strong grasp of big data technologies (e.g., Spark, Delta Lake) and modern data architecture principles.
• Proficient in machine learning lifecycle management using tools like MLflow, scikit-learn, and Spark MLlib.
• Deep understanding of data modeling, warehousing, and SQL, with experience in both traditional and cloud-native environments.
• Skills in Python and SQL, with the ability to write clean, maintainable, and efficient code.
• Knowledge of data governance, quality frameworks, and security best practices.
• Familiarity with containerization (Docker) and orchestration (Kubernetes) for scalable deployments.
• Strong analytical and problem-solving skills, with a consulting mindset for identifying opportunities and delivering value.
• Excellent communication and stakeholder management skills, with experience presenting technical concepts to non-technical audiences.
• Bachelor's or Master's degree in Computer Science, Engineering, or a related field.

Nice to Have

• Experience with data lakes and ML workflows.
• Knowledge of data governance and security frameworks.
• Familiarity with containerization (Docker) and orchestration (Kubernetes).

Qualifications

• Bachelor's or Master's degree in Computer Science, Engineering, or a related field.

Benefits & Perks

• Highly competitive benefits, including medical, dental, and vision insurance.
• A 401(k) plan.
• Tuition reimbursement.
• A work culture focused on innovation and creation of lasting value for clients and employees.