Description

Senior Data Engineer in Bangalore to build scalable data pipelines and enable machine learning workflows using Databricks, Python, and PySpark.

Responsibilities

• Design and maintain scalable ETL/ELT pipelines using Python, PySpark, SQL, and Delta Lake on Databricks.
• Build data ingestion workflows from APIs, product telemetry, and external systems to create ML-ready datasets.
• Operationalize ML models by building feature pipelines and integrating predictions into production data workflows.
• Develop lightweight data applications (Streamlit) and tools to expose insights to internal stakeholders.
• Implement CI/CD best practices, automated workflows, and monitoring for pipeline reliability and cost efficiency.

Required Skills

• 6+ years of experience in data engineering or ML engineering.
• Strong programming skills in Python and SQL for large-scale data processing.
• Extensive experience building scalable data pipelines using Spark or PySpark.
• Hands-on experience with Databricks, Delta Lake, or similar distributed data platforms.
• Experience supporting machine learning workflows and operationalizing ML outputs in production.
• Experience integrating data pipelines with external systems via APIs or automated workflows.
• Strong debugging, monitoring, and performance optimization skills for distributed data pipelines.
• Bachelor’s degree in a relevant field.

Preferred Skills

• Experience with Azure Cloud services including Azure ML, Data Factory, AKS, ACR, and Event Hub.
• Hands-on experience with CI/CD pipelines, Docker, and Kubernetes.
• Familiarity with MLOps practices, model monitoring, and retraining workflows.

Key Skills

Education

Bachelor’s degree