Description

You will design and build scalable data platforms on Azure.

Responsibilities

• Lead the architecture and implementation of data pipelines using Apache Spark, Delta Lake, and Databricks Workflows.
• Design and optimize ETL/ELT processes for structured and unstructured data across Azure, AWS, and GCP.
• Implement real-time streaming and batch processing using Structured Streaming and Delta Live Tables (DLT).
• Integrate ML flow and support deployment of Machine Learning models in production.
• Drive performance tuning and cost optimization of Spark workloads.

Required Skills

• 10+ years of professional experience.
• Proficiency in Azure Databricks and Apache Spark.
• Strong command of Python and SQL programming.
• Experience with Delta Lake implementation.
• Knowledge of Machine Learning concepts and deployment.
• Experience with cloud platforms (Azure, AWS, GCP).
• Familiarity with data security and compliance standards.

Key Skills

Education

Not specified