Description

You will build and maintain scalable data pipelines to ensure data availability for analytical purposes.

Responsibilities

• Design, develop, and maintain ETL/ELT pipelines using Informatica and PySpark.
• Automate workflows and orchestrate complex pipelines using Apache Airflow.
• Migrate existing Informatica ETL processes to PySpark.
• Write complex SQL queries to extract, clean, and transform data.
• Optimize data processes for performance and scalability in cloud or hybrid environments.
• Monitor and troubleshoot data issues to ensure integrity and security.
• Document pipeline designs, data processes, and system architecture.

Required Skills

• 7+ years of hands-on experience in data engineering roles.
• Strong proficiency with Informatica for ETL development.
• Advanced programming skills in Python and experience using PySpark for big data processing.
• Proven experience with Apache Airflow for workflow orchestration.
• Expertise in writing complex SQL queries.
• Experience implementing CI/CD processes and version control.
• Solid understanding of data engineering principles and data modeling.
• Experience writing well-documented, clean, and effective code.
• Degree in BE, B.Tech, MS, M.Tech, or ME from a reputed institute.

Preferred Skills

• Experience with cloud platforms including AWS, GCP, or Azure.
• Knowledge of Kubernetes.
• AWS Solutions Architect certification.

Key Skills

Education

Any Graduate