Lead Data Engineer with experience in Pyspark, Python

About the Role

Key Responsibilities

• Provide technical expertise and coordinate for day-to-day deliverables for the team.
• Assist in the technical design of large business systems.
• Build applications and interfaces between applications.
• Understand data security, retention, and recovery processes.
• Research technologies independently to recommend appropriate solutions.
• Contribute to technology-specific best practices and standards.
• Contribute to success criteria from design through deployment, including reliability, cost-effectiveness, performance, data integrity, maintainability, and scalability.
• Guide and mentor the team during the build and test phases.

Requirements

• Bachelor's degree or foreign equivalent from an accredited institution, or three years of progressive experience in the specialty in lieu of every year of education.
• At least 4 years of Information Technology experience.
• Experience in creating data engineering pipelines using Python with Object-oriented concepts, Pyspark, and AWS cloud native solutions.
• Design, develop, and implement new features to existing frameworks using PySpark and Python.
• Write efficient and effective standalone scripts in PySpark with transformations as per business logic.
• Experience in CI/CD pipeline using Jenkins, CloudFormation, etc.

Nice to Have

• Work closely with the team to understand requirements and develop solutions.
• Test and debug code to ensure desired results.
• Document all programming tasks and procedures for future reference and troubleshooting.
• Use expertise in Python and Object-oriented concepts to solve complex problems and implement robust solutions.
• Proficient in Python, PySpark, and Object-oriented programming concepts.
• Proven experience as a Senior Data Engineer or similar role.
• Strong problem-solving techniques and troubleshooting skills.
• Experience with AWS (preferred but not mandatory).
• Excellent written and verbal communication skills.
• Self-motivated and able to work independently with minimal supervision.
• Ability to work in a diverse/multiple stakeholder environment.
• Experience and desire to work in a global delivery environment.
• Ability to learn and adapt to emerging technologies.

Qualifications

• Bachelor's degree or foreign equivalent from an accredited institution.
• Three years of progressive experience in the specialty in lieu of every year of education.