Big Data Engineer

About the Role

Key Responsibilities

• Design and implement robust data pipelines to extract, transform, and load (ETL) data from various sources, optimizing for efficient AI model training.
• Gather and understand data requirements, create and maintain automated ETL processes with special focus on data flow, error recovery, and exception handling and reporting.
• Support data and cloud transformation initiatives.
• Support our software engineers and data scientists.
• Understand the latest technologies in a rapidly innovative marketplace.
• Ability to work independently and with a team with all stakeholders across the organization to deliver enhanced functionality.

Requirements

• Bachelor's degree in computer science, or similar technical field.
• Minimum 5 years of experience in Big Data Engineering/Data Analysis.
• Proficiency in Python, SQL and Apache Spark.
• Experience with AWS services such as EMR, Glue, S3, Athena, IAM, Lambda and CloudWatch.
• Core Spark, Spark Streaming, DataFrame API, Data Set API, RDD APIs & Spark SQL programming dealing with processing terabytes of data.
• Advanced SQL using Hive/Impala framework including SQL performance tuning.
• Expertise in Hadoop and other distributed computing frameworks.
• Experience with ElasticSearch (OpenSearch) and Kibana Dashboards.
• Knowledge of resource management frameworks such as Yarn or Mesos.
• Experience with physical table design in Big Data environments.
• Familiarity with external job schedulers such as Autosys, AWS Data Pipeline, Airflow.
• Experience working with Key/Value data stores such as HBase.

Qualifications

• Minimum 5 years of experience in Big Data Engineering/Data Analysis.
• Bachelor's degree in computer science or a similar technical field.

Benefits & Perks

• Versatile health perks, including flexible spending accounts, HSA.
• A 401(k) Plan with company match.
• Employee Stock Purchase Plan (ESPP).
• Career opportunities.
• Flexible time away plan.