Description

Design, develop, and maintain scalable data pipelines using Apache Spark and Java.

Responsibilities

• Implement data processing workflows and ETL processes to ingest, transform, and store large volumes of data.
• Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and deliver data solutions.
• Optimize and tune data processing jobs for performance and cost-efficiency.
• Ensure data quality, integrity, and security across all data pipelines and storage solutions.
• Monitor and troubleshoot data pipeline issues, ensuring high availability and reliability.

Required Skills

• 5+ years of professional experience.
• Expertise in Apache Spark and Java development.
• Hands-on experience with AWS services, specifically S3, EMR, Lambda, and Glue.
• Proficiency in ETL Processes implementation.
• Strong command of SQL and NoSQL databases.
• Experience working with Snowflake.
• Familiarity with CI/CD practices.

Key Skills

Education

Any Graduate