Description

You will own the design, development, and maintenance of scalable data pipelines and ELT processes using Python, PySpark, and Apache Airflow.

This role is on-site.

Responsibilities

• Build and maintain data pipelines, workflow orchestration, and custom connectors for structured and unstructured data ingestion.
• Implement DataOps principles and data CI/CD pipelines to automate integration, transformation, and deployment.
• Monitor, troubleshoot, and validate data pipelines to ensure accuracy, consistency, and compliance.
• Collaborate with cross-functional teams to translate business requirements into technical data solutions.
• Document workflows, processes, and technical specifications to support data governance and knowledge sharing.

Required Skills

• 8+ years of experience in data engineering, ELT development, and data modeling.
• Proficiency in Python, PySpark, SQL, and database optimization.
• Hands-on experience with Apache Airflow for workflow orchestration.
• Experience with Trino, Hive, and Snowflake.
• Strong understanding of distributed systems and large-scale datasets.
• Familiarity with data visualization tools like Apache Superset.
• Experience implementing data CI/CD pipelines and DataOps practices.
• Knowledge of version control (Git) and code review processes.

Preferred Skills

• Experience with Linux, OpenShift, and Kubernetes.
• Familiarity with AI concepts and data streaming technologies like Apache Kafka.

Key Skills

Education

Any Graduate