Databricks Data Engineer

About the Role

Key Responsibilities

• Design, develop, and implement robust data solutions using modern data platforms like Databricks.
• Ensure scalable, reliable, and secure data environments that support advanced analytics and meet business needs.
• Lead the migration of data from traditional RDBMS systems to Databricks environments.
• Design scalable data pipelines and infrastructure to meet evolving data demands.
• Develop and manage ETL processes using Databricks for efficient data extraction, transformation, and loading.
• Optimize ETL workflows to improve performance and ensure data integrity.
• Ensure seamless data transitions with minimal disruption to ongoing operations.
• Monitor and optimize data system performance, reliability, scalability, and cost-effectiveness within Databricks environments.
• Collaborate with data engineers, scientists, analysts, and product managers to define and deliver data solutions.
• Establish and enforce best practices and standards for data engineering and data warehouse processes.
• Evaluate and implement tools and technologies to improve data pipeline efficiency and effectiveness.
• Mentor junior data engineers and provide technical leadership.
• Design and operationalize end-to-end machine learning pipelines on Databricks, including data wrangling, feature engineering, training, and tuning.
• Work with DevOps and infrastructure teams to deploy and manage production data systems.
• Ensure compliance with organizational, industry, and regulatory data standards.
• Collaborate with enterprise architects and IT leaders to align data architecture with overall IT strategy.

Requirements

• BS or MS in Computer Science or a related field.
• Extensive experience with Databricks, including ETL and data migration.
• Databricks certifications are a strong plus.
• Experience with cloud platforms like AWS, Azure, or Google Cloud Platform.
• Familiarity with big data technologies and frameworks.
• Experience implementing and optimizing ML workflows using Databricks ML, MLflow, scikit-learn, Spark MLlib, etc.
• Strong knowledge of data warehousing, modeling, and SQL.
• Experience transitioning from RDBMS systems to cloud-native platforms.
• Proficiency in Python, SQL, and scripting languages.
• Understanding of data governance, data quality, and data security best practices.
• Expertise in database technologies, particularly SQL Server.
• Familiarity with Docker and orchestration tools like Kubernetes.
• Excellent analytical and problem-solving skills.
• Strong communication and collaboration skills, especially in cross-functional environments.

Nice to Have

• Databricks certifications.
• Experience with cloud platforms like AWS, Azure, or Google Cloud Platform.