Lead Data Engineer - Big Data

About the Role

Key Responsibilities

• Design & build ingestion pipelines using ELT and schema on read in Databricks delta lake
• Design & develop data transformations using ELT frameworks to modernize ingestion pipelines and build data transformations at scale
• Provide technical expertise in the design and implementation of Ratings Data Ingestion pipelines with modern AWS cloud and other technologies such as S3, Hive, Databricks, Scala, Python, and large-scale data analytics tools
• Build and maintain a data environment for speed, accuracy, consistency, and high availability
• Work closely with other data teams and Data Science team to develop ingestion pipelines
• Ensure data governance principles, data quality checks, and data lineage are implemented at each stage of data processing
• Stay informed about emerging trends in Big Data and cloud technologies and participate in evaluating new technologies
• Ensure compliance with enterprise standards and promote best practices and guiding principles

Requirements

• BE, MCA, or MS degree in Computer Science or Information Technology
• 10+ years of experience as a Data Engineer in an innovative organization
• 5+ years of hands-on experience with data lake systems using AWS/Azure cloud technologies such as S3, Databricks, Hive, and large-scale data analytics tools
• 3+ years of experience building application pipelines using data stream processing tools like Golden Gate for bulk and incremental data loads
• Strong expertise (preferably 5 years) in SDLC cycle including stakeholder management, risk management, and team leadership
• Experience with development frameworks and data/integration technologies such as Python and Scala
• Experience in microservices and API design, including RESTful and SOAP APIs
• Hands-on experience developing scalable data pipelines with Data stream processing tools, Databricks, and Scala applying ETL and ELT concepts
• Deep experience with three or more technologies among Java/J2EE, C#, AWS, large-scale data analytics tools, Python, Scala, RDBMS, Data stream processing tools, Informatica, Angular/ReactJS, cloud-native orchestration tools
• Experience with CI/CD tools like Jenkins and Azure DevOps
• Experience working in UNIX/Linux environments including shell scripting
• Strong understanding of cloud-native architectures, design patterns, and best practices
• Ability to articulate and convert requirements into technical solutions
• Knowledge of industry trends and ability to develop and present technical recommendations
• Familiarity with Agile software development methodologies
• Strong analytical, interpersonal, communication skills, and ability to influence stakeholders

Nice to Have

• Experience with Machine Learning Libraries and Frameworks such as TensorFlow and MLlib
• Financial services industry experience
• Ability to prioritize and manage work in a fast-paced environment
• Experience monitoring industry trends and developing technical recommendations for senior management

Qualifications

• Educational background in Computer Science or Information Technology (BE, MCA, MS)

Benefits & Perks

• Health & Wellness: Health care coverage designed for the mind and body
• Flexible Downtime: Generous time off to stay energized
• Continuous Learning: Access to resources for career growth and skill development
• Invest in Your Future: Competitive pay, retirement planning, continuing education, company-matched student loan contributions, and financial wellness programs
• Family Friendly Perks: Benefits for partners and children
• Beyond the Basics: Retail discounts, referral incentives, and other perks