Big Data Engineer - Massive Data

About the Role

Key Responsibilities

• Data Pipeline Development: Design, develop, and maintain our massive data ingestion pipelines to ensure continuous and reliable flow of image/video/GIS data into our cloud based datastores.
• Workflow Orchestration: Implement and manage data workflows using Apache Airflow to automate, schedule, and monitor data pipelines, ensuring efficient data processing and timely delivery.
• Data Storage and Management: Architect and manage scalable and efficient cloud data storage solutions (preferably GCP).
• Data Integration: Work closely with data scientists, operational analysts, and other stakeholders to integrate data from various sources, including video/GIS, ensuring data consistency, quality, and reliability.
• Performance Optimization: Optimize data pipelines and storage solutions for performance, scalability, and cost-efficiency.
• Security and Compliance: Implement and enforce data security measures, ensuring compliance with industry standards and regulations.
• Automation and Monitoring: Develop automation scripts and monitoring tools to ensure the smooth operation of data pipelines and quickly identify and resolve issues.
• Collaboration: Work closely with the cross functional teams to understand data requirements.
• API Development: Build and maintain RESTful APIs to enable seamless access to data, ensuring reliability, high availability and security.
• Documentation: Maintain comprehensive documentation of data engineering processes, architectures, and best practices.

Requirements

• BS in computer science or equivalent degree with at least 10 years of experience as a software engineer in data engineering work on revenue generating projects.
• Significant experience with Python, Java/Scala programming languages.
• Experience with programming GIS/Map Engines.
• Several years of creating/supporting APIs and throughput optimization.
• Good understanding of data structures, multi-threaded programming and other core computer science fundamentals.
• Experience architecting and implementing cloud native distributed systems with a focus on massive data (100s of TB of data per month).
• Requires sitting or standing at will while performing work on a computer (or any other physical requirements).

Nice to Have

• Understanding of Kubernetes and deployment of distributed data pipelines.
• Understanding of machine learning fundamentals.

Benefits & Perks

• A great Team and culture - please see our Recruitment Video.
• An exciting career as an integral part of a world-leading software company providing solutions for architecture, engineering, and construction.
• Competitive Salary and benefits.
• The opportunity to work within a global and diversely international team.
• A supportive and collaborative environment.
• Colleague Recognition Awards.