Description

You will engineer and manage data processing solutions using big data frameworks.

This role is hybrid.

Responsibilities

• Implement and maintain ETL pipelines and data transformation processes.
• Troubleshoot and resolve issues within PySpark applications and workflows.
• Integrate PySpark code with ingestion frameworks and tools like DataLens.
• Document code lineage and track data flow across converted PySpark code.
• Ensure data quality, integrity, and compliance with security regulations.

Required Skills

• 5+ years of experience in big data development using Hadoop, Hive, and Spark.
• Strong proficiency in Python and PySpark development.
• Expertise in SQL for data querying and relational database interaction.
• Experience designing and implementing ETL pipelines.
• Hands-on knowledge of distributed computing systems, including Spark.
• Familiarity with Kafka and other big data technologies.
• Understanding of data warehousing concepts and CI/CD pipelines.

Preferred Skills

• Experience with SAS.
• Certification in big data or cloud technologies.

Key Skills

Education

Any Graduate