Description

Design, develop, and maintain scalable data pipelines using Databricks, Apache Spark, and Python.
Build and manage feature engineering frameworks and reusable feature pipelines for machine learning use cases.
Develop and maintain Feature Stores to enable training and inference consistency.
Implement ETL/ELT processes for structured and unstructured data from multiple sources.
Optimize Spark jobs, Databricks workloads, and data models for performance and cost efficiency.
Collaborate with Data Scientists, ML Engineers, and Business stakeholders to translate business requirements into data solutions.
Ensure data quality, governance, security, and compliance standards are met.
Design and implement batch and near real-time data processing solutions.
Mentor junior engineers and contribute to architecture and design discussions.
Support CI/CD, automated testing, and deployment of data engineering solutions.
Required Skills

5+ years of experience in Data Engineering.
Strong hands-on experience with Databricks and Apache Spark (PySpark/Scala Spark).
Extensive experience in Feature Engineering for Machine Learning use cases.
Strong programming skills in Python.
Expertise in SQL and data modeling.
Experience with Delta Lake, Databricks Workflows, Unity Catalog, and Lakehouse Architecture.
Experience with cloud platforms such as Azure, AWS, or GCP.
Knowledge of data orchestration tools (Airflow, ADF, etc.).
Understanding of MLOps concepts and ML lifecycle management.
Experience working with large-scale distributed datasets.
Preferred Qualifications

Experience with Databricks Feature Store and MLflow.
Exposure to real-time processing using Kafka/Event Hub.
Knowledge of predictive analytics, recommendation systems, or customer analytics.
Databricks certifications are highly preferred.
Experience in Retail, Client, Healthcare, Banking, or Supply Chain analytics domains

Key Skills

Education

Bachelor's degree