Description

You will design, build, and optimize scalable data pipelines to support analytics, business intelligence, and machine learning.

Responsibilities

• Design, develop, and maintain ETL pipelines using AWS, Python, and Spark.
• Optimize data ingestion, transformation, and storage processes for efficient data processing.
• Manage structured and unstructured data while ensuring integrity, quality, and governance.
• Develop SQL queries to extract and manipulate data from relational databases.
• Implement data validation and testing frameworks using Pytest to ensure accuracy.
• Monitor and troubleshoot pipelines to minimize downtime.

Required Skills

• 10+ years of experience in Data Engineering or a related field.
• Strong proficiency in AWS services including S3, Glue, Lambda, EMR, and Redshift.
• Expertise in Python for data processing and automation.
• Hands-on experience with Apache Spark for distributed processing.
• Deep understanding of ETL pipeline design and data warehousing concepts.
• Proficiency in SQL for managing relational databases.
• Experience writing unit and integration tests using Pytest.
• Familiarity with Git and CI/CD pipelines.
• Any Graduate degree.

Preferred Skills

• Experience with Terraform, Docker, or Kubernetes.
• Knowledge of Apache Kafka or Airflow.
• Exposure to data governance and security best practices.

Key Skills

Education

Any Graduate