Description

You will design and develop data integration solutions using Ab Initio ETL and Apache Spark, managing complex workflows across hybrid environments.

This role is on-site.

Responsibilities

• Design and develop scalable data pipelines using Ab Initio and Apache Spark.
• Create, optimize, and maintain Directed Acyclic Graphs (DAGs) in Python for workflow orchestration.
• Leverage AWS services including S3, Athena, Glue, and EMR for data lifecycle management.
• Support migration efforts from PySpark to AWS-based solutions.
• Maintain technical documentation including HLD, LLD, and mapping specifications.

Required Skills

• 5+ years of experience in ETL development.
• Expertise in Ab Initio.
• Proficiency in Python, Java, and ANSI SQL.
• Hands-on experience with Apache Spark and distributed computing.
• Experience with Airflow for workflow orchestration and DAG optimization.
• Solid understanding of AWS services (S3, Athena, Glue, EMR).
• Ability to write HLD, LLD, and mapping documents.

Preferred Skills

• Experience with data migration from PySpark to AWS.
• Familiarity with cloud monitoring and automation tools.
• Exposure to data governance and lifecycle management.

Key Skills

Education

Any Gradute