Description

You will design and maintain scalable data architectures and ETL pipelines within an AWS environment.

Responsibilities

• Design, develop, and maintain ETL pipelines using AWS services, Python, and Spark.
• Optimize data ingestion, transformation, and storage processes for high-performance processing.
• Implement data validation and testing frameworks using Pytest to ensure accuracy and reliability.
• Monitor and troubleshoot data pipelines to ensure smooth operation and minimal downtime.
• Collaborate with data scientists, analysts, and software engineers to build scalable data solutions.

Required Skills

• 5+ years of experience in Data Engineering or a related field.
• Strong proficiency in AWS (S3, Glue, Lambda, EMR, Redshift).
• Hands-on experience with Python for data processing and automation.
• Expertise in Apache Spark for distributed data processing.
• Proficiency in SQL for querying and managing relational databases.
• Experience writing unit and integration tests using Pytest.
• Solid understanding of ETL pipeline design and data warehousing concepts.
• Familiarity with CI/CD pipelines and version control systems like Git.

Preferred Skills

• Experience with Terraform, Docker, or Kubernetes.
• Knowledge of big data tools such as Apache Kafka or Airflow.
• Exposure to data governance and security best practices.

Key Skills

Education

Any Gradute