Description

You will design and maintain scalable data pipelines on AWS, owning the lifecycle from ingestion to analysis. As an individual contributor, you will build robust ETL/ELT workflows and real-time streaming systems while ensuring data integrity and performance across platforms.

Responsibilities

• Design and develop scalable data pipelines using AWS Glue, Glue Catalog, Amazon S3, Athena, and AWS Lambda.
• Build and optimize ETL/ELT workflows using PySpark and Python for large-scale data processing.
• Develop and manage real-time data streaming pipelines using Apache Kafka to ensure low latency and high reliability.
• Create, maintain, and optimize SQL queries for data extraction, transformation, and analysis.
• Troubleshoot, debug, and optimize data workflows to meet performance and scalability requirements.

Required Skills

• 5+ years of experience in data engineering or related fields.
• Proficiency in Python and PySpark for data processing.
• Hands-on experience with AWS Glue, AWS Lambda, and Glue Catalog.
• Strong knowledge of Amazon S3, AWS Athena, and SQL.
• Experience building ETL pipelines and handling batch data.
• Familiarity with Apache Kafka for streaming data ingestion.
• Bachelor's degree in Computer Science, Engineering, or a related field.

Preferred Skills

• Experience with Glue Streaming for advanced real-time processing.
• Familiarity with AWS CDK for infrastructure as code implementations.

Key Skills

Education

Any Graduate