Description

You will design and maintain data pipelines using Azure Databricks for ingestion, transformation, and loading into various data stores.

Responsibilities

• Implement data processing solutions using Apache Spark for distributed computing and real-time processing.
• Develop and optimize data models and schemas for efficient storage and retrieval.
• Collaborate with data scientists and analysts to translate data needs into technical solutions.
• Define and implement data governance policies for quality and security.
• Monitor and troubleshoot pipelines on Azure Databricks to ensure performance and reliability.

Required Skills

• 10+ years of experience in data engineering.
• Strong proficiency in Python or Scala for data manipulation.
• Hands-on experience with Apache Spark core concepts, data structures, and APIs.
• Deep experience with Azure Databricks, including workspace management, cluster configuration, and job scheduling.
• Experience with ETL and ELT ingestion and transformation techniques.
• Proficiency with Azure Data Lake Storage, Azure Blob Storage, and Azure SQL Database.
• Experience with Azure Synapse Analytics.
• Knowledge of data warehousing and data lake concepts.
• Understanding of data security principles and best practices.

Key Skills

Education

Any Graduate