Description

Design, develop, and maintain scalable data pipelines and feature engineering frameworks for machine learning use cases.

This role is on-site.

Responsibilities

• Build and manage reusable feature pipelines and Feature Stores to ensure training and inference consistency.
• Develop and maintain ETL/ELT processes for structured and unstructured data using Databricks, Apache Spark, and Python.
• Optimize Spark jobs, Databricks workloads, and data models for performance and cost efficiency.
• Implement batch and near real-time data processing solutions while ensuring data quality, governance, and security standards.
• Mentor junior engineers and support CI/CD, automated testing, and deployment of data engineering solutions.

Required Skills

• 5+ years of experience in Data Engineering.
• Strong hands-on experience with Databricks and Apache Spark (PySpark/Scala Spark).
• Extensive experience in Feature Engineering for Machine Learning use cases.
• Strong programming skills in Python with expertise in SQL and data modeling.
• Experience with Delta Lake, Databricks Workflows, Unity Catalog, and Lakehouse Architecture.
• Experience with cloud platforms such as Azure, AWS, or GCP.
• Knowledge of data orchestration tools like Airflow or ADF.
• Understanding of MLOps concepts and ML lifecycle management.
• Experience working with large-scale distributed datasets.

Preferred Skills

• Experience with Databricks Feature Store and MLflow.
• Exposure to real-time processing using Kafka or Event Hub.
• Databricks certifications.

Key Skills

Education

Bachelor's degree