Description

You will architect and implement scalable data solutions on Databricks, Apache Spark, and Delta Lake across Azure and AWS environments.

Responsibilities

• Lead migration initiatives from legacy systems to Databricks and cloud-native platforms.
• Design and optimize ETL workflows, data lakes, and real-time streaming pipelines.
• Establish data governance, security, and compliance standards for engineering teams.
• Collaborate with cross-functional teams to define data strategies and integration frameworks.
• Provide technical leadership and architectural guidance to support engineering delivery.

Required Skills

• 14-20 years of overall experience in data engineering or architecture.
• Proven expertise with Databricks, Apache Spark, and Delta Lake.
• Hands-on experience with Azure and AWS cloud platforms.
• Strong knowledge of data warehousing, ETL processes, and advanced analytics.
• Experience with MLflow and machine learning environment optimization.
• Bachelor’s or Master’s degree in Computer Science, Data Engineering, or related field.
• Excellent problem-solving and communication skills for technical leadership.

Preferred Skills

• Experience with CI/CD pipelines and DevOps practices for data engineering.
• Knowledge of containerization (Docker, Kubernetes) and microservices architecture.
• Familiarity with BI tools like Power BI and Tableau for data visualization.

Key Skills

Education

Bachelor's or Master's degrees