Description

You will design, build, and maintain scalable data pipelines to collect and process data from diverse sources.

This role is remote.

Responsibilities

• Design and maintain efficient data pipelines using SparkSQL and Scala to ingest user interactions, financial details, and external feeds.
• Build scalable pipelines leveraging the Airflow scheduler and executor framework.
• Develop data models that support merchandising optimization while ensuring data quality and consistency.
• Improve code and data quality by contributing to internal tools that detect and mitigate issues.
• Collaborate with Data Scientists, Product Managers, and Software Engineers to define requirements and deliver data solutions.

Required Skills

• 5-9+ years of industry experience with a BS/Masters, or 2+ years with a PhD.
• Expert level proficiency in SQL and Python.
• Intermediate experience with Spark and Scala.
• Hands-on experience with distributed processing frameworks like Hadoop and Kafka.
• Experience with ETL schedulers such as Apache Airflow, Luigi, Oozie, or AWS Glue.
• Solid understanding of data warehousing and relational databases like PostgreSQL and MySQL.
• Experience with columnar databases such as Redshift, BigQuery, HBase, or ClickHouse.
• Experience with distributed storage systems like HDFS or S3.

Preferred Skills

• Ability to analyze large datasets to identify gaps, inconsistencies, and insights.

Key Skills

Education

Any Graduate