Description

You will design, build, and maintain scalable data pipelines and feature engineering frameworks for machine learning use cases.

This role is on-site.

Responsibilities

• Develop and optimize ETL/ELT processes and Spark jobs using Databricks and Apache Spark.
• Build and manage Feature Stores and reusable pipelines to ensure training and inference consistency.
• Implement batch and near real-time data processing solutions for structured and unstructured data.
• Collaborate with Data Scientists and ML Engineers to translate business requirements into robust data solutions.
• Ensure data quality, governance, security, and compliance while mentoring junior engineers.

Required Skills

• 5+ years of experience in Data Engineering with strong Python programming skills.
• Hands-on expertise with Databricks, Apache Spark (PySpark/Scala), and Delta Lake.
• Deep understanding of Lakehouse Architecture, Unity Catalog, and Databricks Workflows.
• Proficient in SQL, data modeling, and optimizing distributed datasets.
• Experience with cloud platforms: Azure, AWS, or GCP.
• Familiarity with data orchestration tools like Airflow or ADF.
• Knowledge of MLOps concepts and ML lifecycle management.

Preferred Skills

• Experience with Databricks Feature Store, MLflow, Kafka, or Event Hubs.
• Databricks certifications and domain expertise in Retail, Healthcare, Banking, or Supply Chain.

Key Skills

Education

Bachelor's degree