Description

You will design, build, and maintain efficient data pipelines to collect and process data from user interactions, financial details, and external feeds.

Responsibilities

• Build scalable data pipelines using SparkSQL and Scala leveraging Airflow.
• Develop data models to enable efficient analysis and merchandising optimization.
• Analyze large data sets to identify gaps, inconsistencies, and provide actionable insights.
• Collaborate with Data Scientists, Product Managers, and Software Engineers to define requirements and deliver data solutions.
• Contribute to internal tooling and standards to improve code quality and data reliability.

Required Skills

• 5-9+ years of relevant industry experience with a BS/Masters, or 2+ years with a PhD.
• Expert level SQL proficiency.
• Intermediate experience with Spark and Scala.
• Strong Python skills.
• Experience with distributed processing frameworks including Hadoop, Spark, and Kafka.
• Experience with distributed storage systems such as HDFS or S3.
• Expertise with ETL schedulers like Apache Airflow, Luigi, Oozie, or AWS Glue.
• Hands-on experience with relational databases such as PostgreSQL or MySQL.
• Experience with columnar databases including Redshift, BigQuery, HBase, or ClickHouse.
• Understanding of data warehousing concepts.

Key Skills

Education

Any Graduate