Sr Data Engineer

About the Role

Key Responsibilities

• Develop, optimize, and maintain Spark applications and jobs for data processing, transformation, storage, and resource management using Scala, Python, SQL, Shell or Java.
• Design, build, and maintain end-to-end data pipelines using Spark for ETL/ELT processes and manage data lake using Databricks and cloud technologies.
• Develop and maintain streaming data pipelines with Apache Kafka and Spark Streaming, implementing robust monitoring and alerting solutions to ensure data integrity and performance.
• Integrate and manage data flows using SOAP and REST APIs for ingestion, transformation, and database interactions, ensuring secure, reliable, and efficient data transfer.
• Collaborate with data scientists, analysts, and stakeholders to deliver data solutions, maintain comprehensive documentation, and continuously improve data engineering processes with the latest big data technologies and best practices.

Requirements

• Bachelor's degree or equivalent in Computer Science, Applied Mathematics, Engineering, or a related field.
• 3 years of related experience working as a big data engineer.
• 2 years of experience developing Big Data pipelines leveraging Spark, Python, Scala, Shell, and SQL.
• Experience designing and operating enterprise-grade data streaming use cases with GCP PubSub, Spark Streaming, or Kafka.
• Experience designing and delivering large-scale, fault-tolerant applications using Spark and cloud services to ensure data integrity.
• Knowledge of data quality frameworks, monitoring mechanisms, and validation rules using Databricks Delta Lake and Apache Spark.
• Experience implementing security measures including secret management and encryption for protecting PII information.
• Familiarity with data privacy laws and regulations including GDPR, SOX, and HIPAA.

Nice to Have

• Experience with cloud technologies such as GCP.
• Knowledge of data security and compliance standards.
• Experience working in a collaborative, fast-paced environment.

Qualifications

• Bachelor's degree or equivalent in a relevant field
• 3 years of experience as a big data engineer

Benefits & Perks

• Pet friendly environment, bring your pets to work!
• On-site Dog Park
• Top Dog gym with equipment, fitness classes, massage therapists, and personal trainers
• Sit & Stay Café serving fresh breakfast and lunch options
• On-site coffee bar
• Lil' Paws learning center and onsite paid daycare facility
• Volunteer events with PetSmart Charities
• Access to PetSmart benefits and wellness programs