Description

You will design and develop optimized data pipelines for both batch and real-time data processing.

Responsibilities

• Design and develop optimized data pipelines for batch and real-time processing.
• Implement data warehousing concepts to support large-scale data storage and retrieval.
• Write and optimize complex SQL queries using Hive and PySpark data frames.
• Optimize joins and processing logic for handling massive datasets.

Required Skills

• 5+ years of experience with Python or Java.
• Deep understanding of Hadoop and Spark architecture and working principles.
• Hands-on experience with SQL, including Hive and PySpark data frames.
• Strong knowledge of data warehousing concepts.
• Proven ability to optimize joins when processing large volumes of data.
• Experience building both batch and real-time data pipelines.
• Any Graduate degree.

Key Skills

Education

Any Graduate