Description

You will design and develop scalable data pipelines using Azure Databricks, PySpark, and Spark SQL.

Responsibilities

• Build and optimize ETL/ELT workflows for ingestion, transformation, and processing of large datasets.
• Implement and manage Delta Lake architectures for batch and streaming data.
• Develop reusable notebooks, jobs, and workflows within Databricks.
• Optimize Spark jobs for performance, scalability, and cost efficiency.
• Implement data quality, validation, monitoring, and governance standards.

Required Skills

• 8+ years of experience in data engineering or big data development.
• Expert-level experience with Azure Databricks, PySpark, and Spark SQL.
• Hands-on experience with Azure Data Lake (ADLS) and Azure Data Factory.
• Strong understanding of Delta Lake, medallion architecture, and distributed data processing.
• Proficiency with SQL and relational databases.
• Experience building batch and streaming data pipelines.
• Familiarity with CI/CD, DevOps, and Git-based workflows.
• Experience with performance tuning and optimizing Spark jobs.
• Strong understanding of data modeling, structured/unstructured data, and metadata management.

Preferred Skills

• Databricks Certified Data Engineer Associate/Professional.
• Experience with Azure Synapse, Cosmos DB, or Event Hub/Kafka.
• Knowledge of Terraform, ARM templates, or IaC tools.

Key Skills

Education

Bachelor's degree