Description

Design and implement end-to-end data processing pipelines and scalable data lake solutions using the Databricks platform.

Responsibilities

• Design batch and streaming data ingestion, transformation, and analysis pipelines.
• Architect scalable data lake solutions to manage structured and unstructured data.
• Develop and optimize Spark jobs, notebooks, and feature engineering workflows.
• Deploy machine learning models using MLflow for predictive analytics.
• Implement data governance, security controls, and compliance protocols.

Required Skills

• 5+ years of experience in a Databricks Architect or similar data analytics role.
• Strong proficiency in Apache Spark and the Databricks platform.
• Hands-on experience with Databricks notebooks, Spark SQL, and DataFrame API.
• Experience with MLflow for model deployment.
• Proficiency in cloud-based platforms including AWS and Azure.
• Deep understanding of data lake architecture principles.
• Knowledge of data governance, security, and compliance requirements.

Preferred Skills

• Databricks certification (Developer or Professional Data Scientist).
• Experience with Docker, Kubernetes, and DevOps CI/CD practices.

Key Skills

Education

Bachelor's degree in Computer Science