Description

Build and maintain data pipelines using Spark and AWS.

Responsibilities

• Analyze existing SQL scripts and map data from new sources to current architectures.
• Integrate SQL and Spark SQL scripts into the existing data pipeline.
• Modify and optimize existing pipelines using Spark and AWS services.
• Perform data analysis and manage connections to various data sources.

Required Skills

• 5+ years of experience in data engineering.
• Advanced SQL proficiency for script analysis and data mapping.
• Strong experience with Python and PySpark.
• Hands-on experience with Spark, Flink, Kafka, or NoSQL technologies.
• Proficiency in Big Data processing and Scala or Java.
• Experience with AWS services including ECS, EC2, EMR, and Lambda.
• Knowledge of CI/CD workflows and Microsoft Azure.

Key Skills

Education

Any Graduate