Description

You will build and maintain data pipelines using Python, PySpark, and SQL for big data processing.

Responsibilities

• Build and manage ETL processes using Informatica.
• Maintain data lakes utilizing the Iceberg format.
• Ensure data quality across all processing stages.
• Develop and automate workflows using Airflow.
• Manage data storage and compute using AWS services.

Required Skills

• 5+ years of experience in data engineering.
• Proficiency in Python and PySpark.
• Strong SQL skills for data manipulation.
• Experience with Informatica for ETL.
• Hands-on experience with AWS services including S3, Glue, Redshift, Lambda, and EMR.
• Knowledge of Postgres and Airflow.
• Ability to write Bash/Shell scripts.
• Experience working with Iceberg data lake formats.
• Any Graduate degree.

Key Skills

Education

Any Graduate