Description

You own the design, development, and maintenance of scalable data pipelines.

Responsibilities

• Design, develop, and maintain ETL pipelines using AWS services, Python, and Spark.
• Optimize data ingestion, transformation, and storage processes for efficient data processing.
• Develop SQL queries to extract and manipulate data from relational databases.
• Implement data validation and testing frameworks using Pytest to ensure data accuracy.
• Monitor and troubleshoot data pipelines to ensure continuous operation.

Required Skills

• 15+ years of experience in Data Engineering or a related field.
• Strong proficiency in AWS (S3, Glue, Lambda, EMR, Redshift, etc.).
• Hands-on experience with Python for data processing and automation.
• Expertise in Apache Spark for distributed data processing.
• Solid understanding of ETL pipeline design and data warehousing concepts.
• Proficiency in SQL for querying and managing relational databases.
• Experience writing unit and integration tests using Pytest.
• Familiarity with CI/CD pipelines and Git.

Preferred Skills

• Experience with Terraform, Docker, or Kubernetes.
• Knowledge of big data tools such as Apache Kafka or Airflow.

Key Skills

Education

Any Graduate