Description

You will design and maintain scalable data pipelines to process user interactions, financial data, and external feeds.

This role is remote.

Responsibilities

• Build scalable data pipelines using SparkSQL and Scala within the Airflow scheduler/executor framework.
• Develop data models to enable efficient analysis and manipulation of data for merchandising optimization.
• Design and maintain pipelines that ensure data quality, consistency, and accuracy across all sources.
• Analyze large data sets to identify gaps, provide insights, and advance product solutions.
• Contribute to internal tooling and standards to improve code quality and automate issue detection.

Required Skills

• Expert-level proficiency in Python and SQL.
• Hands-on experience with Spark and Scala.
• Deep expertise with Apache Airflow.
• Experience with distributed processing frameworks including Hadoop, Spark, and Kafka.
• Experience with distributed storage systems such as HDFS or S3.
• Solid understanding of data warehousing and relational databases like PostgreSQL and MySQL.
• Experience with columnar databases such as Redshift, BigQuery, HBase, or ClickHouse.
• Experience with ETL schedulers like Luigi, Oozie, or AWS Glue.

Preferred Skills

• 5+ years of relevant industry experience with a BS/Masters, or 2+ years with a PhD.
• Collaboration with Data Scientists, Product Managers, and Software Engineers.

Key Skills

Education

Any Graduate