Description

What You'll Do:

• Architect, design, and develop scalable batch and streaming data pipelines using Azure Databricks, PySpark, Python, and Kafka
• Build and optimize enterprise-grade ETL/ELT workflows for high-volume, complex data processing
• Design and implement bronze, silver, and gold data layers using Delta Lake and medallion architecture
• Develop reusable Databricks notebooks, Python modules, and PySpark frameworks for ingestion, transformation, validation, and orchestration
• Design Kafka-based streaming solutions and integrate real-time data sources with the Azure data platform
• Establish data governance using Unity Catalog, including catalog and schema management, lineage, auditing, and controlled data access
• Implement role-based access control using Microsoft Entra ID and define secure access policies across data assets
• Create scalable conceptual, logical, and physical data models aligned with analytics and business requirements
• Lead performance tuning, production troubleshooting, code reviews, deployment practices, and technical collaboration across engineering teams

Expertise You'll Bring:

• 8 to 12 years of experience in Data Engineering, including ownership of complex enterprise data solutions
• Strong hands-on expertise in Azure Databricks, Apache Spark, PySpark, Delta Lake, and medallion architecture
• Extensive experience in ETL/ELT pipeline design, development, orchestration, optimization, and production support
• Strong proficiency in Python and SQL, including reusable development, complex transformations, and query optimization
• Hands-on experience with Kafka and real-time or streaming data processing
• Strong experience in data governance using Unity Catalog, access control with Microsoft Entra ID, and enterprise data modeling
• Experience with Azure Data Factory, ADLS, Git, CI/CD, Azure DevOps, data quality, monitoring, and security practices

Key Skills

Education

Any Graduate