Data/Analytics Engineer (768025)

About the Role

Key Responsibilities

• Build and maintain ETL/ELT pipelines using Azure Databricks and Microsoft Fabric.
• Implement medallion architecture (Bronze, Silver, Gold layers) to support data lifecycle and quality.
• Develop real-time and batch ingestion processes from IES Gateway and other source systems.
• Ensure data quality, validation, and transformation logic is consistently applied.
• Use Python, Spark, and SQL in Databricks and Fabric notebooks for data transformation.
• Implement Delta Lake for data versioning, ACID transactions, and schema enforcement.
• Integrate Databricks with other Azure services like Azure One Lake, Azure ADLS Gen2, and Microsoft Fabric.
• Collaborate with Domain Owners to design dimensional and real-time data models.
• Create analytics-ready datasets for Power BI and other reporting tools.
• Standardize field naming conventions and schema definitions across datasets.
• Apply data classification and tagging based on DECAL's data governance framework.
• Implement row-level security, data masking, and audit logging as per compliance requirements.
• Support integration with Microsoft Purview for lineage and metadata management.
• Partner with BI developers to ensure data models are optimized for Power BI.
• Provide curated datasets that align with reporting requirements and business logic.
• Create BI dashboards and train users.
• Support CI/CD pipelines for data workflows using Azure DevOps.
• Assist in monitoring, logging, and performance tuning of data jobs and clusters.

Requirements

• Bachelor's degree in computer science, Data Engineering, or related field.
• 3+ years of experience in data engineering or analytics engineering roles.
• Proficiency in advanced SQL techniques for data transformation, querying, and optimization.
• Hands-on experience with Azure Databricks (Spark, Delta Lake).
• Experience with Microsoft Fabric (Dataflows, Pipelines, OneLake).
• Knowledge of SQL and Python (Pandas, PySpark).
• Experience with SQL Server 2019+.
• Familiarity with data modeling, data governance, and data security best practices.
• Strong understanding of ETL/ELT processes, data quality, and schema design.

Nice to Have

• Experience with Power BI datasets and semantic modeling.
• Knowledge of Microsoft Purview, Unity Catalog, or similar governance tools.
• Exposure to real-time data processing and streaming architectures.
• Knowledge of federal/state compliance requirements for data handling.
• Familiarity with Azure DevOps, Terraform, or CI/CD for data pipelines.
• Certifications such as Microsoft Fabric Analytics Engineer.

Qualifications

• Educational background as specified in requirements.