Description

You will design, develop, and maintain large-scale data pipelines and analytical solutions on Azure.

This role is remote.

Responsibilities

• Design and optimize scalable data pipelines using Azure Databricks and ADF for ingestion, transformation, and integration.
• Develop and maintain ETL/ELT workflows using PySpark, ensuring high performance and data accuracy.
• Implement data quality, validation, and governance processes across data pipelines.
• Monitor, debug, and improve pipeline performance and reliability using Azure tools.
• Collaborate with data architects and stakeholders to translate data requirements into technical solutions.

Required Skills

• 6+ years of experience in Data Engineering or related roles.
• Strong hands-on experience in Azure Databricks (Spark, Delta Lake, notebooks, cluster management).
• Expertise in PySpark for distributed data processing and optimization.
• Proficiency in Azure Data Factory (ADF) for pipeline orchestration.
• Solid understanding of data modeling, data warehousing, and ETL best practices.
• Experience with Azure Data Lake (ADLS Gen2), Azure Synapse Analytics, and SQL-based systems.
• Strong SQL skills for complex queries and performance tuning.
• Experience with JSON, Parquet, Avro, and Delta data formats.
• Knowledge of CI/CD pipelines for data solutions using Azure DevOps or Git.

Preferred Skills

• Experience with Power BI.
• Exposure to Machine Learning pipelines using Databricks MLflow.
• Knowledge of Databricks REST APIs and automation scripts.

Key Skills

Education

Any Graduate