Description

Design and implement scalable data processing pipelines using Apache Spark. Optimize Spark jobs for transformation, aggregation, and analysis. Ensure data quality, integrity, and consistency across all pipelines.

Responsibilities

• Design and implement scalable data processing pipelines using Apache Spark.
• Develop and optimize Spark jobs for data transformation, aggregation, and analysis.
• Extract, process, and analyze data from various sources within large datasets.
• Implement data integration solutions to connect disparate data sources.
• Monitor and troubleshoot performance issues in Spark jobs and cluster environments.

Required Skills

• 5+ years of experience in data engineering.
• Proficiency with Apache Spark using Java or PySpark.
• Strong SQL skills for querying, managing, and manipulating datasets.
• Experience with Azure cloud platforms.
• Knowledge of cloud-based data storage, processing, and deployment.
• Ability to ensure data quality, integrity, and consistency across pipelines.
• Experience documenting technical designs, processes, and procedures.

Preferred Skills

• Any Graduate degree.

Key Skills

Education

Any Graduate