Description

You will design and implement scalable data pipelines to ingest, transform, and store structured and unstructured data.

Responsibilities

• Build and maintain ETL/data pipelines using cloud technologies and Databricks.
• Develop metadata-driven data ingestion pipelines using ADF.
• Analyze, optimize, and tune existing pipelines for performance, reliability, and efficiency.
• Implement MLOps practices to streamline machine learning model deployment and management.
• Orchestrate workflows and job scheduling using Apache Airflow.

Required Skills

• 10+ years of experience in Data Engineering.
• 4+ years of experience designing scalable pipelines with Databricks.
• 5+ years of experience with Python, PySpark, and SQL.
• 5+ years of experience building metadata-driven ingestion pipelines using ADF.
• 2+ years of experience with Apache Airflow for orchestration.
• 2+ years of experience implementing MLOps practices.
• Familiarity with CI/CD tools for automating data engineering deployments.
• Strong understanding of ETL processes, data architecture, and security compliance.
• Ability to translate business requirements into technical data solutions.

Key Skills

Education

Any Gradute