Description

Design, implement, and optimize scalable data pipelines for analytics, business intelligence, and machine learning.

This role is hybrid.

Responsibilities

• Design and maintain ETL pipelines using AWS, Python, and Spark.
• Optimize data ingestion, transformation, and storage processes.
• Develop SQL queries to extract and manipulate data from relational databases.
• Implement data validation and testing frameworks using Pytest to ensure accuracy.
• Monitor and troubleshoot pipelines to ensure minimal downtime.

Required Skills

• 5+ years of experience in Data Engineering or a related field.
• Proficiency in AWS services including S3, Glue, Lambda, EMR, and Redshift.
• Hands-on experience with Python for data processing and automation.
• Expertise in Apache Spark for distributed data processing.
• Strong proficiency in SQL for managing relational databases.
• Experience writing unit and integration tests using Pytest.
• Solid understanding of ETL pipeline design and data warehousing.
• Familiarity with CI/CD pipelines and version control systems like Git.

Preferred Skills

• Experience with Terraform, Docker, or Kubernetes.
• Knowledge of Apache Kafka or Airflow.
• Exposure to data governance and security best practices.

Key Skills

Education

Any Graduate