Description

You will build and manage scalable data pipelines using Apache Spark and Java.

Responsibilities

• Design, develop, and maintain scalable data pipelines using Apache Spark and Java.
• Implement ETL processes and data processing workflows to ingest, transform, and store large datasets.
• Optimize and tune data processing jobs for performance and cost-efficiency.
• Monitor and troubleshoot pipeline issues to ensure high availability and reliability.
• Develop and maintain data models, schemas, and documentation while ensuring data quality and security.

Required Skills

• 5+ years of experience in data engineering or related roles.
• Proficiency in Java and Apache Spark.
• Hands-on experience with AWS services, specifically S3, EMR, Lambda, and Glue.
• Experience with Snowflake.
• Strong knowledge of SQL and NoSQL databases.
• Experience with CI/CD tools, including Jenkins and Spinnaker.
• Ability to collaborate with data scientists and analysts to meet data requirements.

Preferred Skills

• Experience with Jules.

Key Skills

Education

Any Graduate