Description

Design, build, and unit test Spark framework applications using Python for batch and streaming data requirements. Optimize Spark application performance in Hadoop by configuring Spark Context, Spark-SQL, DataFrames, and Pair RDDs. Develop and execute data pipeline testing processes to validate business rules and policies. Design and implement data solutions across platforms including Hadoop, Teradata, Ab Initio, and Snowflake.

Responsibilities

• Build PySpark-based applications to handle batch and streaming data requirements.
• Optimize Spark application performance in Hadoop by configuring Spark Context, Spark-SQL, DataFrames, and Pair RDDs.
• Develop and execute data pipeline testing processes to validate business rules and policies.
• Design and implement data solutions across platforms including Hadoop, Teradata, Ab Initio, and Snowflake.
• Design and test applications on the Spark framework.

Required Skills

• 5+ years of experience in data engineering or related roles.
• Expertise in PySpark and Python development.
• Hands-on experience with Hadoop and NoSQL databases.
• Proficiency with SQL and SSIS.
• Experience with AWS Lake Formation and S3.
• Practical knowledge of Teradata Data Warehouse, Ab Initio, and Informatica.
• Experience working with Java Spark (DPL) and Snowflake.

Key Skills

Education

ANY GRADUATE