Description

You will build and scale data platforms to enable data products, business analytics, and data science.

Responsibilities

• Design and maintain end-to-end data ingestion pipelines from raw sources to Big Data repositories.
• Develop real-time data pipelines and event streaming systems to simplify data access for stakeholders.
• Extract, transform, and load structured and unstructured data to support analytics and feature generation.
• Manage data lake and Hadoop platform infrastructure, including performance monitoring and system modifications.
• Collaborate with data stewards to ensure data accuracy, lineage, and effective governance.

Required Skills

• 5+ years of relevant industry experience in big data engineering.
• Proficiency in Python, Scala, Java, or SQL.
• Hands-on experience with Spark, Kafka, and Hive.
• Proven ability to build high-volume data ingestion and streaming pipelines using Kafka, AWS Kinesis, or Spark Streaming.
• Experience designing, building, and scaling production-ready event streaming systems.
• Expertise in building data lake or data warehouse solutions for structured and unstructured data.
• Strong knowledge of SQL query optimization, including partitioning, bucketing, and indexing.
• Technical familiarity with Hadoop ecosystem tools such as PySpark, Yarn, HBase, Cloudera, NiFi, MapReduce, and Linux.
• Bachelor's degree in Computer Science or a related discipline.

Preferred Skills

• Experience with advanced technologies such as H2O and ElasticSearch.

Key Skills

Education

Bachelor's degree in Computer Science