Description

Design, build, and optimize PySpark applications for batch and streaming data pipelines on Hadoop and cloud platforms.

Responsibilities

• Build and unit test Spark framework applications using Python for batch and streaming requirements.
• Optimize Spark performance in Hadoop by configuring Spark Context, Spark-SQL, DataFrames, and Pair RDDs.
• Develop and execute data pipeline testing processes to validate business rules and policies.
• Design and implement data solutions across Hadoop, Teradata, Ab Initio, and Snowflake.
• Design and test applications on the Spark framework.

Required Skills

• 5+ years of experience in data engineering or related roles.
• Expertise in PySpark and Python development.
• Hands-on experience with Hadoop and NoSQL databases.
• Proficiency with SQL and SSIS.
• Experience with AWS Lake Formation and S3.
• Practical knowledge of Teradata Data Warehouse, Ab Initio, and Informatica.
• Experience working with Java Spark (DPL) and Snowflake.

Key Skills

Education

ANY GRADUATE