Description

You will develop and maintain scalable data pipelines for large-scale data processing.

Responsibilities

• Design and develop scalable data pipelines using Apache Spark.
• Process large datasets utilizing Spark Core, Spark SQL, Spark Streaming, and Spark MLlib.
• Develop ETL workflows for both structured and unstructured data.
• Optimize Spark jobs specifically for performance and cost efficiency.
• Implement real-time data processing using Apache Kafka.

Required Skills

• 5+ years of professional experience.
• Strong experience with Apache Spark using Scala, Python, or Java.
• Proficiency in PySpark.
• Hands-on experience with Hadoop and HDFS.
• Working knowledge of SQL and NoSQL databases.
• Experience with cloud platforms (AWS, Azure, or GCP).
• Familiarity with workflow orchestration tools, preferably Airflow.
• Understanding of data warehousing concepts.

Key Skills

Education

Any Graduate