Description

Build end-to-end data pipelines from the ground up, transitioning from a software engineering foundation to large-scale data systems.

Responsibilities

• Build and maintain end-to-end data pipelines.
• Transform business requirements into production code, analytical reports, and tools.
• Perform root cause analysis, investigation, debugging, and testing of data systems.
• Document technical requirements, data lineage, and subject matter for business and technical audiences.
• Collaborate with product managers and engineering teams to provide business insights.

Required Skills

• 5+ years of engineering experience with a software engineering background.
• Proficiency in Java or Python for application development.
• Strong programming skills in Java, Scala, Flume, or Python.
• Experience with distributed computing frameworks including Hadoop, Hive, Pig, or Apache Spark.
• Hands-on experience with Spark or MapReduce/YARN.
• Advanced SQL skills, specifically with Teradata physical design, implementation, and performance optimization.
• Experience processing petabytes of data using Hadoop scripting.
• Proficiency in Unix/Linux shell scripting.
• Experience with ETL processes and real-time data ingestion using Kafka.

Preferred Skills

• Experience working within the retail business sector.

Key Skills

Education

Bachelor’s Degree