Description

You will design and implement ETL pipelines using Microsoft Fabric and Azure Data Factory. You will build a metadata-driven Lakehouse architecture with threaded datasets to support multiple consumption patterns. You will develop agent-specific data lakes and an orchestration layer for an uber agent that queries across agents to answer customer questions. You will enable interactive data consumption via Power BI, Azure OpenAI, and other analytics tools. You will ensure data quality, lineage, and governance across all ingestion and transformation processes. You will collaborate with product teams to deliver scalable solutions and optimize performance and cost across storage and compute layers.

Responsibilities

• Design and implement ETL pipelines using Microsoft Fabric (Dataflows, Pipelines, Lakehouse, Warehouse, SQL) and Azure Data Factory.
• Build and maintain a metadata-driven Lakehouse architecture with threaded datasets to support multiple consumption patterns.
• Develop agent-specific data lakes and an orchestration layer for an uber agent that queries across agents to answer customer questions.
• Enable interactive data consumption via Power BI, Azure OpenAI, and other analytics and AI tools.
• Ensure data quality, lineage, and governance across all ingestion and transformation processes.

Required Skills

• 7+ years of experience in data engineering with a focus on Microsoft Azure Data stack and Fabric technologies.
• Strong expertise in Microsoft Fabric (Lakehouse, Dataflows Gen2, Pipelines, Notebooks).
• Proficiency in Azure Data Factory, Azure SQL, and Azure Data Lake Storage Gen2.
• Experience with Power BI and/or other visualization tools.
• Strong command of SQL, Python, and PySpark/Scala.
• Experience working with structured and semi-structured data (JSON, XML, CSV, Parquet).
• Proven ability to build metadata-driven architectures and reusable components.
• Strong understanding of data modeling, data governance, and security best practices.

Preferred Skills

• Familiarity with agent-based architectures and conversational AI integration.

Key Skills

Education

Any Graduate