Big Data Engineer - IV

About the Role

Key Responsibilities

• Lead in design, development, and testing of data ingestion pipelines, perform end-to-end validation of ETL processes for various datasets being ingested into the big data platform.
• Perform data migration and conversion validation activities on different applications and platforms.
• Provide technical leadership on data profiling, analysis, discovery, and assessment of data quality issues within data sources.
• Contribute to development of transformation logic, interfaces, and reports to meet project requirements.
• Participate in discussions on technical architecture, data modeling, and ETL standards, collaborating with Product Managers, Architects, and Senior Developers.
• Lead in designing and developing validation frameworks and automated test suites to validate data pipeline flow, transformation rules, and data integrity.
• Develop tools to measure data quality and visualize anomaly patterns in source and processed data.
• Assist the Manager in project planning and validation strategy development.
• Support User Acceptance Testing and production validation activities.
• Provide technical recommendations for data validation tools and new technologies to improve validation processes.
• Evaluate existing methodologies and processes, and recommend improvements.
• Work with stakeholders, Product Management, Data and Design, Architecture teams, and executives to identify issues and contribute to resolutions.

Requirements

• 8+ years of software development and testing experience.
• 4+ years of experience with tools like Spark, HBase, Hive, Sqoop, Impala, Kafka, Flume, Oozie, MapReduce.
• 4+ years of programming experience in Scala, Java, or Python.
• Experience in technical leadership and mentoring teams.
• Experience developing and testing ETL, real-time data-processing, and analytics applications.
• Strong knowledge of Spark SQL, Scala development in Hadoop environment, and BI/DW development.
• Strong knowledge of shell scripting.
• Experience with Web Services API development and testing.
• Experience with CI/CD automation frameworks.
• Experience with cloud environments such as AWS or GCP is a plus.
• Knowledge of GIT/Jenkins and pipeline automation.
• Understanding of software development practices and tools.
• Strong analytical and problem-solving skills in Big Data domain.
• Good organizational, written, and verbal communication skills.

Nice to Have

• Working experience on large migration projects.
• Experience with Google Cloud Platform.
• Development experience for monitoring and alert tools.
• Familiarity with project management and bug tracking tools like JIRA.

Qualifications

• Bachelor's Degree in computer science or engineering.

Benefits & Perks

• Pay range of $50-57 per hour, based on experience and qualifications.
• Medical & pharmacy coverage.
• Dental and vision insurance.
• 401(k) plan.
• Health savings account (HSA) and Flexible Spending Account (FSA).
• Life Insurance, Pet Insurance, Short and Long Term Disability.
• Accident & Critical Illness coverage.
• Pre-paid legal & ID theft protection.
• Sick time and other paid leaves as required by law.
• Employee Assistance Program (EAP).