Description

You will build and manage data pipelines and streaming architectures within an AWS environment.

Responsibilities

• Design and manage data pipelines using AWS Glue, EMR, and Kafka.
• Develop streaming data solutions using K-SQL and Kafka topics.
• Handle database storage and retrieval mechanisms for MongoDB and DocuDB.
• Collaborate with business stakeholders to translate evolving requirements into technical solutions.
• Guide the team through technical implementation to meet project milestones.

Required Skills

• 5+ years of experience as a PySpark Developer or in a related data engineering role.
• 8+ years of total IT development experience.
• Proficiency in Python, Scala, or Spark.
• Hands-on experience with AWS services including S3, Athena, Glue, EMR, and VPC.
• Experience with Kafka, K-SQL, and streaming data platforms.
• Strong knowledge of NoSQL databases, specifically MongoDB and AWS DocuDB.
• Experience with relational databases such as PostgreSQL.
• Proficiency in Unix, FTP, and file handling.
• Experience working within Agile methodologies using JIRA and Confluence.

Preferred Skills

• Experience with Apache NiFi.
• Experience with Hudi.

Key Skills

Education

ANY GRADUATE