AWS Pyspark Data Engineer

About the Role

Key Responsibilities

• Develop and maintain data pipelines using AWS Glue Studio and AWS Glue ETL to ensure efficient data processing and integration
• Use AWS Glue Catalog to lead and catalog metadata for data lakes and data warehouses
• Implement data storage solutions using Amazon S3 to ensure secure and scalable data management
• Leverage Apache Spark for large-scale data processing and analytics to drive business insights
• Collaborate with multi-functional teams to understand data requirements and deliver solutions that meet business needs
• Optimize data workflows and processes to improve performance and reduce costs
• Provide technical mentorship and support to junior developers and team members
• Ensure data quality and integrity through detailed testing and validation processes
• Supervise and tackle data pipeline issues to ensure timely resolution and minimal impact on operations
• Stay updated with the latest industry trends and technologies to continuously improve skills and knowledge
• Participate in the development of best practices and standards for data engineering and analytics
• Participate in code reviews and provide constructive feedback to improve code quality
• Document data workflows, processes, and solutions to ensure clarity and knowledge sharing

Requirements

• Expertise in Python, AWS Glue Studio, AWS Glue ETL, AWS Glue Catalog, Amazon S3, and Apache Spark
• Experience in the Cards & Payments domain is a plus
• Certifications: AWS Certified Developer, AWS Certified Solutions Architect, Apache Spark Certification
• Ability to develop and maintain data pipelines and data storage solutions
• Strong collaboration skills to work with multi-functional teams
• Knowledge of data quality, testing, and validation processes
• Ability to troubleshoot and resolve data pipeline issues
• Continuous learning to stay updated with industry trends and technologies

Nice to Have

• Experience in the Cards & Payments domain

Qualifications

• Certifications: AWS Certified Developer, AWS Certified Solutions Architect, Apache Spark Certification

Benefits & Perks

• Medical/Dental/Vision/Life Insurance
• Paid holidays plus Paid Time Off
• Long-term/Short-term Disability
• Paid Parental Leave
• Employee Stock Purchase Plan