Description

Key Skills: Pyspark, Azure Databricks, Azure ADF, SQL, Azure Data lake

Roles and Responsibilities:

• Design and implement data pipelines using Pyspark and Azure Databricks.
• Develop and maintain Azure Data Factory (ADF) workflows for data integration.
• Collaborate with data scientists and analysts to understand data requirements and deliver solutions.
• Optimize data storage and processing using Azure Data Lake and other Azure services.
• Create and manage Databricks notebooks using Python for data processing tasks.
• Ensure data quality and integrity through rigorous testing and validation processes.
• Monitor and troubleshoot data pipeline performance and reliability.

Skills Required:

• Strong hands-on experience in Pyspark for large-scale data processing.
• Proven expertise in Azure Databricks for data engineering and transformation workflows.
• Experience in building and managing data pipelines using Azure Data Factory (ADF).
• Good understanding of Azure Data Lake storage architecture and optimization techniques.
• Strong SQL skills for data querying and performance tuning.
• Experience handling structured and unstructured data sources.
• Knowledge of data quality frameworks and validation techniques.
• Familiarity with performance monitoring and troubleshooting of data pipelines.
• Strong analytical and problem-solving skills.

Education: Any Graduation, Any Post Graduation in Computer Science

Key Skills

Education

Any Graduate