GCP Data Engineer

About the Role

Key Responsibilities

• Work in an implementation team from concept to operations, providing deep technical subject matter expertise for successful deployment.
• Implement methods for automation of all parts of the pipeline to minimize labor in development and production.
• Analyze complex data, organize raw data, and integrate massive datasets from multiple data sources to build subject areas and reusable data products.
• Collaborate with architects to evaluate and productionalize appropriate GCP tools for data ingestion, integration, presentation, and reporting.
• Work with stakeholders to formulate business problems as technical data requirements, identify and implement technical solutions, and ensure key business drivers are captured.
• Design and deploy pipelines with automated data lineage, and develop, evaluate, and summarize Proof of Concepts to validate solutions.
• Test and compare competing solutions and report on the best options.
• Design and build production data engineering solutions to deliver pipeline patterns using GCP services such as BigQuery, DataFlow, Pub/Sub, BigTable, Data Fusion, DataProc, Cloud Composer, Cloud SQL, Compute Engine, Cloud Functions, and App Engine.

Requirements

• Experience working in an implementation team from concept to operations, providing deep technical expertise.
• Experience in analyzing complex data, organizing raw data, and integrating datasets from multiple sources.
• Experience working with architects to evaluate and productionalize GCP tools for data ingestion, integration, and reporting.
• Experience in formulating business problems as technical data requirements and implementing solutions.
• Designing and deploying data pipelines with automated data lineage.
• Proficiency with GCP services such as BigQuery, DataFlow, Pub/Sub, BigTable, Data Fusion, DataProc, Cloud Composer, Cloud SQL, Compute Engine, Cloud Functions, and App Engine.
• In-depth understanding of Google's product technology or other cloud platforms.
• At least 5+ years of analytics application development experience.
• At least 5+ years of SQL development experience.
• 3+ years of Cloud experience (GCP preferred) with solutions designed and implemented at production scale.
• 2+ years of professional development experience in Java or Python, and Apache Beam.
• Experience with extracting, loading, transforming, cleaning, and validating data.
• Designing data processing pipelines and architectures.
• 1+ year of designing and building CI/CD pipelines.

Nice to Have

• Experience building Machine Learning solutions using TensorFlow, BigQueryML, AutoML, Vertex AI.
• Experience in building solution architecture, provisioning infrastructure, and securing data-centric services in GCP.
• Experience with DataPlex or Informatica EDC.
• Experience with development ecosystems such as Git, Jenkins, and CI/CD.
• Experience working with DBT/Dataform.
• Experience working with Agile and Lean methodologies.
• Performance tuning experience.
• GCP Professional Data Engineer Certification.
• Master's degree in computer science or related field.
• Mentoring experience with engineers (2+ years).

Qualifications

• Bachelor's degree in computer science or related scientific field.
• IT or related certifications and topics such as data architect, data center, data integrity, data manager, data management, data scientist, data warehousing, SQL, Sybase, Teradata.