Senior Data Engineer - GCP

About the Role

Key Responsibilities

• Design, develop, and optimize data pipelines on GCP using BigQuery, Dataflow, and Pub/Sub
• Build and maintain microservices to support data processing and analytics
• Develop real-time and batch data ingestion frameworks using Kafka
• Work with NoSQL databases (e.g., Firestore, Cassandra, MongoDB) to support high-throughput data applications
• Ensure data integrity, governance, and security best practices in the data pipeline
• Optimize performance, scalability, and cost efficiency of data solutions
• Collaborate with data scientists, analysts, and software engineers to deliver end-to-end data products
• Implement monitoring, logging, and alerting for data pipelines and services
• Participate in agile development methodologies, attend meetings, contribute to project planning, and help drive project success
• Explore new tools, frameworks, and techniques to improve digital solutions and foster innovation
• Create and maintain technical documentation, including system designs, architecture diagrams, and code documentation, while sharing knowledge with team members

Requirements

• 5+ years of overall experience in large-scale software development with at least 2+ years in a technical lead role
• 5+ years of experience in data engineering focusing on microservices-based data solutions
• 5+ years of designing and developing highly scalable Microservices, Webservices, and Rest APIs
• 3+ years of hands-on experience with GCP services like BigQuery, Dataflow, Pub/Sub, and Cloud Storage
• 3+ years of containerization (Docker, Kubernetes) and CI/CD for data pipelines
• 3+ years of experience developing within an agile development environment

Nice to Have

• Experience designing and developing in a Test-Driven Development environment
• Proficiency in Python or Java for data processing
• Experience with monitoring and logging tools like Prometheus, Stackdriver, or ELK
• Understanding of distributed computing and data partitioning strategies
• Experience with containerization and orchestration technologies (e.g., Docker, Kubernetes)
• Experience with Terraform or Infrastructure as Code (IaC)
• Familiarity with Apache Beam or Flink for large-scale data processing
• Knowledge of machine learning pipelines and integration with data engineering workflows
• Experience in performance tuning and cost optimization in cloud environments
• Experience with new SQL databases

Qualifications

• Bachelor's degree or equivalent experience (HS diploma + 4 years relevant experience)

Benefits & Perks

• Affordable medical plan options
• A 401(k) plan including matching company contributions
• An employee stock purchase plan
• No-cost programs including wellness screenings, tobacco cessation, weight management, confidential counseling, and financial coaching
• Paid time off, flexible work schedules, family leave, dependent care resources
• Colleague assistance programs, tuition assistance, retiree medical access
• Comprehensive and competitive pay and benefits package