Description

You will design and implement production-quality data pipelines within a distributed systems environment.

Responsibilities

• Build and maintain efficient, high-quality data pipelines using Java, Scala, or Python.
• Manage distributed systems and workflows using Hadoop, Spark, and Airflow or Oozie.
• Deploy and manage data workloads on AWS services including EC2, EMR, and RDS.
• Collaborate with cross-functional teams using Agile methodologies and test-driven development.
• Implement automated testing and source control management to ensure pipeline reliability.

Required Skills

• 12+ years of experience in data engineering or related software engineering roles.
• 5+ years of production-quality coding experience in Java, Scala, or Python.
• In-depth experience with Hadoop/MapReduce, Spark, Hive, and Kafka.
• Proficiency with workflow orchestration tools like Airflow or Oozie.
• Hands-on experience with AWS cloud services: EC2, EMR, and RDS.
• Experience with GIT, JIRA, Jenkins, and Shell scripting.
• Strong understanding of test-driven development and test automation.
• Bachelor’s degree in Computer Science, Informatics, Information Systems, or equivalent work experience.

Preferred Skills

• Experience building Marketing Data pipelines, including Direct Mail.
• Experience with Snowflake and Salesforce Marketing Cloud.
• Knowledge of open-source ML frameworks and end-to-end model development life cycles.
• Experience running containers (Docker/LXC) using Kubernetes, AWS ECS, or AWS EKS.

Key Skills

Education

Any Graduate