Description

You will lead the design and implementation of scalable data pipelines and lakehouse solutions within the Azure ecosystem.

Responsibilities

• Design, develop, and maintain scalable ETL processes and data pipelines using Azure Databricks and Data Factory.
• Implement and optimize Spark jobs, data transformations, and complex workflows.
• Develop data models, data dictionaries, and integrate data from diverse sources into Databricks.
• Establish data quality checks, governance policies, and security procedures.
• Monitor pipeline performance and optimize workflows to ensure data consistency and reliability.

Required Skills

• 5+ years of experience in data engineering, data pipelines, and data integration.
• Proficiency in Azure Databricks and PySpark.
• Strong programming skills in Python, SQL, and Scala.
• Deep understanding of Spark programming and job optimization.
• Hands-on experience with ETL processes and data warehousing concepts.
• Expertise in data modeling and data dictionary development.
• Working knowledge of Microsoft Azure cloud platforms.
• Proven ability in data governance, security procedures, and problem-solving.
• Any Graduate degree.

Key Skills

Education

Any Graduate