Description

You design and implement scalable data processing pipelines using Apache Spark to transform, aggregate, and analyze large datasets.

This role is on-site.

Responsibilities

• Develop and optimize Spark jobs for data transformation and analysis.
• Implement data integration solutions to connect disparate sources.
• Monitor and troubleshoot performance issues in Spark jobs and cluster environments.
• Ensure data quality, integrity, and consistency across all pipelines.
• Document technical designs, processes, and procedures.

Required Skills

• 5+ years of experience in data engineering.
• Proficiency with Apache Spark using Java or PySpark.
• Strong SQL skills for querying and manipulating datasets.
• Experience with Azure cloud platforms.
• Knowledge of cloud-based data storage, processing, and deployment.
• Ability to ensure data quality and consistency.

Key Skills

Education

Any Graduate