Description

You will lead the design and implementation of scalable data pipelines and lakehouse solutions within the Azure ecosystem.

Responsibilities

• Design, develop, and maintain scalable ETL processes and data pipelines using Azure Databricks and Data Factory.
• Implement and optimize Spark jobs, data transformations, and workflows.
• Develop and maintain data models, data dictionaries, and data quality checks.
• Integrate data from diverse sources into Azure Databricks and manage data warehouse deliveries.
• Collaborate with data scientists and analysts to ensure data consistency and implement governance and security procedures.

Required Skills

• 5+ years of experience in data engineering and data integration.
• Proficiency with Azure Databricks and PySpark.
• Strong programming skills in Python, SQL, and Scala.
• Expertise in Spark programming and job optimization.
• Deep understanding of ETL processes, data warehousing concepts, and data modeling.
• Hands-on experience with Microsoft Azure cloud services.
• Proven ability in data governance, security procedures, and quality monitoring.
• Strong debugging and problem-solving skills.
• Bachelor's degree or equivalent experience.

Key Skills

Education

Any Graduate