Description

You will design and optimize scalable data pipelines, streaming solutions, and data models across enterprise platforms.

Responsibilities

• Design and implement streaming data ingestion pipelines using Kafka producers, consumers, and topics.
• Manage partitions, consumer groups, offsets, and retention strategies for scalability and fault tolerance.
• Build and optimize ETL/ELT workflows in Dataiku and Spark on Cloudera.
• Write scalable Python/PySpark scripts for data cleansing, transformation, and enrichment.
• Integrate Kafka with downstream platforms such as Spark, Cloudera, or Snowflake.

Required Skills

• 3–5 years of professional experience in analytics, data science, or business intelligence roles.
• Proficiency in Kafka for real-time streaming implementation.
• Experience developing and optimizing Spark jobs using partitioning, broadcast joins, and caching.
• Strong skills in Python and PySpark for data transformations.
• Expertise in SQL for data manipulation.
• Experience with Snowflake as a downstream platform.
• Familiarity with data pipeline design and monitoring.
• Bachelor's Degree in Engineering, Statistics, Mathematics, Computer Science, Data Science, Economics, or a related quantitative field.

Key Skills

Education

Bachelor's degree