Description

You will design and build data architecture using advanced data engineering techniques.

Responsibilities

• Design and implement metadata-driven solutions for data ingestion.
• Implement data ingestion patterns, focusing on batch processing for CSV files.
• Apply data modeling standards, such as the Medallion architecture (Bronze, Silver, Gold).
• Process and transform data using PySpark within Fabric Spark environments.
• Troubleshoot data issues and assess data requirements.

Required Skills

• 8+ years of experience building data architecture.
• In-depth knowledge of Microsoft Fabric components: Lakehouse, OneLake, Delta Lake, Fabric Pipelines, Dataflows Gen2.
• Strong understanding of data lake concepts and optimal OneLake usage.
• Proficiency in PySpark for data processing and transformation.
• Advanced SQL skills for querying and database management.
• Experience implementing data ingestion patterns (batch processing).
• Familiarity with Medallion architecture.
• Experience with Data Pipeline Performance monitoring.
• Strong analytical and problem-solving capabilities.

Preferred Skills

• Understanding of data governance principles and data security measures.
• Knowledge of monitoring data pipeline performance.

Key Skills

Education

Any Graduate