Description

You will design, build, and maintain efficient data pipelines to collect and process data from user interactions, financial details, and external feeds.

Responsibilities

• Build scalable data pipelines using SparkSQL and Scala within the Airflow scheduler/executor framework.
• Develop data models to enable efficient analysis and manipulation of data for merchandising optimization.
• Design and maintain pipelines that ensure data quality, consistency, and accuracy across all sources.
• Analyze large data sets to identify gaps, provide insights, and advance product solutions.
• Collaborate with Data Scientists, Product Managers, and Software Engineers to define requirements and deliver data solutions.
• Contribute to internal tooling and standards to improve code quality and automate issue detection.

Required Skills

• Expert-level proficiency in Python and SQL.
• 5+ years of relevant industry experience with a BS/Masters, or 2+ years with a PhD.
• Hands-on experience with Spark and Scala.
• Deep expertise with Apache Airflow.
• Experience with distributed processing frameworks including Hadoop, Spark, and Kafka.
• Experience with distributed storage systems such as HDFS or S3.
• Solid understanding of data warehousing and relational databases like PostgreSQL and MySQL.
• Experience with columnar databases such as Redshift, BigQuery, HBase, or ClickHouse.
• Experience with ETL schedulers like Luigi, Oozie, or AWS Glue.

Key Skills

Education

Any Graduate