Description

You design, develop, and maintain scalable data pipelines using Databricks, Apache Spark, and Python.

This role is on-site.

Responsibilities

• Build and manage feature engineering frameworks and reusable pipelines for machine learning use cases.
• Develop and maintain Feature Stores to ensure training and inference consistency.
• Implement ETL/ELT processes for structured and unstructured data from multiple sources.
• Optimize Spark jobs, Databricks workloads, and data models for performance and cost efficiency.
• Support CI/CD, automated testing, and deployment of data engineering solutions.

Required Skills

• 5+ years of experience in Data Engineering.
• Strong hands-on experience with Databricks and Apache Spark (PySpark/Scala Spark).
• Extensive experience in Feature Engineering for Machine Learning use cases.
• Strong programming skills in Python.
• Expertise in SQL and data modeling.
• Experience with Delta Lake, Databricks Workflows, Unity Catalog, and Lakehouse Architecture.
• Experience with cloud platforms such as Azure, AWS, or GCP.
• Knowledge of data orchestration tools (Airflow, ADF, etc.).
• Understanding of MLOps concepts and ML lifecycle management.

Preferred Skills

• Experience with Databricks Feature Store and MLflow.
• Exposure to real-time processing using Kafka/Event Hub.
• Databricks certifications are highly preferred.

Key Skills

Education

Any Gradute