Description

You will design and maintain ETL pipelines and data infrastructure on Azure.

This role is on-site.

Responsibilities

• Develop ETL pipelines using Python and PySpark to extract, transform, and load data from various sources.
• Build configuration-driven ETL jobs using YAML files to simplify maintenance and deployment.
• Design data models for both relational (SQL Server) and NoSQL (MongoDB) databases.
• Deploy ETL applications on Azure Kubernetes Service (AKS) and manage Docker containers.
• Establish CI/CD pipelines for automated deployments and monitor pipeline performance in production.

Required Skills

• 10+ years of experience in data engineering or backend development.
• Strong proficiency in Python for data transformations and business logic.
• Hands-on experience with PySpark for large-scale data processing.
• Deep knowledge of SQL, including complex queries, optimization, and performance tuning on SQL Server.
• Experience with NoSQL databases, specifically MongoDB.
• Proficiency in containerization with Docker and orchestration with Kubernetes (YAML configurations).
• Experience setting up and maintaining CI/CD pipelines.

Key Skills

Education

Any Gradute