Sr Data Engineer

About the Role

Key Responsibilities

• Design and Develop ETL Pipelines: Architect, implement, and optimize ETL processes to handle the ingestion, transformation, and storage of large-scale IoT data streams. Ensure data accuracy, consistency, and availability for analysis.
• Create and Maintain Datasets: Develop and manage comprehensive datasets that aggregate IoT data from various sources. Ensure datasets are well-structured, accessible, and aligned with business needs.
• Data Integration and Optimization: Integrate disparate data sources and optimize data storage and retrieval processes to support efficient querying and reporting. Implement best practices for data quality and performance.
• Collaborate with Stakeholders: Work closely with business users, data scientists, and analysts to understand data requirements and translate them into technical solutions. Provide support and troubleshooting for data-related issues.
• Monitor and Improve Data Systems: Continuously monitor the performance of data pipelines and datasets. Identify and address potential issues, and implement improvements to enhance efficiency and reliability.
• Documentation and Compliance: Document data processes, pipeline architectures, and system changes. Ensure compliance with data governance and security policies.

Requirements

• Minimum of 4 years of professional experience in data analysis or data engineering.
• Experience with Big Data/Hadoop applications.
• Experience with Spark, Hive, Kafka, and Map Reduce.
• Experience building production data pipelines.
• Experience with different data formats (Parquet, Avro, Json, XML, etc.).
• Experience bringing in data from multiple Data Sources / API's.
• Building and Debugging Data Pipelines.
• Knowledge of distributed systems (pipelines and databases).
• Proficient in Python development for data pipelines and automation, including debugging, testing, and handling development. At least 2 years experience preferred.
• Experience in Linux environments, batch automation, and shell scripts.
• Graduate Degree in a Quantitative field (Math, Statistics, Engineering, Computer Science) or relevant experience.
• 3+ years work experience.

Nice to Have

• Familiarity with cloud platforms (e.g., AWS, Azure, GCP) and data warehousing solutions (e.g., Redshift, Snowflake).
• Experience working with IoT data.

Benefits & Perks

• Paid holidays and flexible paid time away.
• 401(k) + Employer Match.
• Paid holidays, medical/dental/vision/life coverage & 24/7 Medical Hotline.
• Onsite health clinic, gym, gaming tables.
• Employee/Friends/Family Discounts.
• Free daily lunch and drinks on site.
• Employee Resource Groups.