Description

You will design and develop scalable data pipelines using Azure Databricks, PySpark, and Spark SQL.

This role is on-site.

Responsibilities

• Build and optimize ETL/ELT workflows for large dataset ingestion and transformation.
• Implement Delta Lake architectures for batch and streaming data processing.
• Develop reusable notebooks, jobs, and workflows within Databricks.
• Optimize Spark jobs for performance, scalability, and cost efficiency.
• Enforce data quality, validation, monitoring, and governance standards.

Required Skills

• 8+ years in data engineering or big data development.
• Expert-level Azure Databricks, PySpark, and Spark SQL.
• Hands-on experience with Azure Data Lake (ADLS) and Azure Data Factory.
• Strong understanding of Delta Lake, medallion architecture, and distributed processing.
• Proficiency in SQL and relational databases.
• Experience building batch and streaming data pipelines.
• Familiarity with CI/CD, DevOps, and Git-based workflows.

Preferred Skills

• Databricks Certified Data Engineer Associate/Professional.
• Experience with Azure Synapse, Cosmos DB, or Event Hub/Kafka.
• Knowledge of Terraform, ARM templates, or IaC tools.

Key Skills

Education

Bachelor's degree