Data Engineer

Sterling Computers

North Sioux City, SD

Full Time

Mid Level

5 days ago

Job Description

About the Role

Sterling is seeking a highly motivated and skilled Data Engineer to join our growing team. In this role, you will be instrumental in building, maintaining, and optimizing our data infrastructure, enabling data-driven decision-making across the organization. If you're passionate about working with cutting-edge technologies, thriving in an agile environment, and collaborating closely with other technical experts, we want to hear from you! As a Data Engineer, you will be a key player in our data ecosystem, responsible for the design, development, and maintenance of robust and scalable data pipelines. You will work with a variety of data sources, ensuring data quality, accessibility, and reliability. This role requires a blend of strong technical skills, an eagerness to learn, and communication abilities to gather requirements and collaborate effectively.

Key Responsibilities

Design, build, deploy, and maintain resilient and efficient data pipelines using Databricks and Python, ensuring data is accurately and timely delivered to various stakeholders.
Lead and contribute to the design and implementation of new data solutions and platforms from inception, bringing innovative approaches to our data architecture.
Actively participate in the modernization of our existing data platforms, enhancing their scalability, efficiency, and capabilities.
Curate, prepare, and provide high-quality datasets for consumption by data analytics teams, data scientists, and machine learning engineers, ensuring data readiness for various analytical and modeling needs with high visibility.
Take ownership of our data catalog, ensuring it is comprehensive, accurate, and up-to-date, facilitating data discovery and governance.
Develop complex SQL queries for data extraction, transformation, and loading (ETL) processes, and optimize existing queries for performance.
Proactively engage with cross-functional teams to understand data needs, gather detailed requirements, and translate them into technical solutions.
Participate actively in our Agile Scrum methodology, including sprint planning, daily stand-ups, and retrospectives, contributing to continuous improvement.
Identify, diagnose, and resolve data-related issues, ensuring data integrity and system stability.
Stay abreast of the latest industry trends and technologies in data engineering, advocating for and implementing best practices to keep our data platform at the cutting edge.

Requirements

Proven experience with Databricks, including Spark, Delta Lake, and related services, for large-scale data processing.
Strong proficiency in Python for data manipulation, scripting, and pipeline development.
Expert-level SQL skills with experience in optimizing complex queries.
Solid understanding and practical experience with Agile Scrum development processes and tools like Jira.
Excellent interpersonal skills with a demonstrated ability to work effectively in a team environment, communicate technical concepts to non-technical audiences, and gather precise requirements.
A strong sense of ownership over data pipelines and data quality, with a proactive approach to problem-solving and maintenance.
Eagerness to learn new technologies, adapt to evolving challenges, and contribute to a fast-paced, innovative data team.

Nice to Have

Experience with data platform modernization and scalability improvements.
Knowledge of data governance and data catalog management best practices.

Qualifications

Experience with large-scale data processing platforms such as Databricks, Spark, and Delta Lake.
Proficiency in Python programming.
Expertise in SQL query development and optimization.
Practical experience with Agile Scrum methodologies and tools like Jira.

Benefits & Perks

A dynamic and collaborative work environment where your contributions make a real impact.
Opportunities to work with state-of-the-art data technologies and learn continuously.
A culture that values innovation, continuous improvement, and professional growth.
Competitive salary and benefits package.