Description

Design, develop, and maintain large-scale data processing pipelines using Hadoop, Spark, Python, and Scala. Implement scalable data ingestion, storage, transformation, and analysis solutions. Optimize existing pipelines for performance and reliability. Monitor production pipelines to troubleshoot issues and ensure data integrity.

Responsibilities

• Design and build distributed data processing systems using Hadoop, Spark, Hive, and Trino.
• Implement efficient data ingestion and transformation workflows in Python, Scala, or Java.
• Optimize SQL queries using window functions and complex joins for better performance.
• Troubleshoot and resolve production issues to maintain continuous data flow.
• Collaborate with cross-functional teams to translate business requirements into technical solutions.

Required Skills

• 5+ years of experience with Big Data technologies: Hadoop, Spark, Hive, Trino.
• Proficiency in Java, Scala, or Python for distributed systems development.
• Strong SQL skills, including optimization, window functions, and complex queries.
• Hands-on experience with AI development tools and workflow design.
• Bachelor's degree in Computer Science, Information Systems, or related field.

Preferred Skills

• Experience with cloud platforms and distributed systems architecture.

Key Skills

Education

Any Graduate