Description

You will design, develop, and maintain large-scale data processing pipelines using distributed systems and cloud platforms. You own the full lifecycle of data ingestion, storage, transformation, and analysis, ensuring scalability and efficiency. You optimize existing pipelines for performance and resolve production issues to maintain data integrity. You translate business requirements into technical solutions and collaborate with cross-functional teams. You also apply AI development tools and workflow design to enhance data capabilities.

Responsibilities

• Design and build scalable data pipelines using Hadoop, Spark, Hive, and Trino.
• Implement data ingestion, transformation, and storage solutions that are efficient and reliable.
• Optimize pipeline performance, scalability, and reliability through code refinement.
• Monitor production pipelines, troubleshoot issues, and ensure continuous data flow.
• Collaborate with teams to translate business requirements into technical data solutions.

Required Skills

• 5+ years of experience with Big Data technologies including Hadoop, Spark, Hive, and Trino.
• Proficiency in object-oriented programming languages: Java, Scala, or Python.
• Deep expertise in distributed systems and cloud platforms.
• Strong SQL skills, including window functions, joins, and complex query optimization.
• Hands-on experience with AI development tools and AI workflow design.
• Bachelor's degree in Computer Science, Information Systems, or a related discipline.

Preferred Skills

• Experience optimizing SQL code for high-performance environments.
• Proven track record of improving pipeline reliability and data integrity.

Key Skills

Education

Any Graduate