Description

Build and optimize scalable data pipelines using PySpark within a distributed big data environment.

This role is hybrid.

Responsibilities

• Develop and maintain ETL processes and data pipelines using PySpark.
• Process large datasets using Spark SQL, DataFrames, and RDDs.
• Perform performance tuning to ensure data integrity and system efficiency.
• Collaborate with data analysts and scientists to meet data requirements.
• Leverage Hadoop ecosystem tools for big data processing.

Required Skills

• 5+ years of experience with PySpark and Python.
• Strong hands-on experience with Spark SQL, DataFrames, and RDDs.
• Proven track record in ETL development and data processing.
• Deep knowledge of the Hadoop ecosystem (HDFS, Hive).
• Understanding of big data architecture and distributed systems.
• Experience with performance tuning for data pipelines.

Preferred Skills

• Experience with cloud platforms (AWS, Azure, or GCP).
• Familiarity with Kafka or other streaming tools.
• Knowledge of data warehousing concepts.

Key Skills

Education

Any Graduate