Description

You will design, build, and maintain large-scale data pipelines and analytical solutions using Azure Databricks and Azure Data Factory.

This role is on-site.

Responsibilities

• Design and optimize scalable data pipelines for ingestion, transformation, and integration.
• Develop and maintain ETL/ELT workflows using PySpark, ensuring high performance and data accuracy.
• Implement data quality, validation, and governance processes across data pipelines.
• Monitor, debug, and improve pipeline performance and reliability using Azure tools.
• Collaborate with data architects and stakeholders to translate data requirements into technical solutions.

Required Skills

• 6+ years of experience in Data Engineering or related roles.
• Strong hands-on experience in Azure Databricks (Spark, Delta Lake, notebooks, cluster management).
• Expertise in PySpark for distributed data processing and optimization.
• Proficiency in Azure Data Factory (ADF) for pipeline orchestration and integration.
• Experience with Azure Data Lake (ADLS Gen2), Azure Synapse Analytics, and SQL-based systems.
• Knowledge of CI/CD pipelines for data solutions using Azure DevOps or Git.
• Strong SQL skills for complex queries and performance tuning.
• Experience with JSON, Parquet, Avro, and Delta data formats.

Preferred Skills

• Experience with Power BI.
• Exposure to Machine Learning pipelines using Databricks MLflow.

Key Skills

Education

Any Graduate