Description

Design and maintain data pipelines and backend systems for large-scale data processing.

This role is on-site.

Responsibilities

• Design, develop, and maintain data pipelines using Spark, Python, Scala, and Java.
• Write optimized SQL queries for ETL processes and manipulate large datasets with DataFrames.
• Build and maintain real-time data streaming pipelines using MSK/Kafka.
• Implement data storage and processing solutions using cloud technologies (AWS or GCP) and S3.
• Ensure data quality, integrity, and security while collaborating with data scientists and engineers.

Required Skills

• 5+ years of backend development experience with a focus on data engineering.
• Strong proficiency in Spark, Python, Scala, and Java.
• Expertise in SQL and working with relational databases.
• Experience with message streaming platforms like MSK/Kafka.
• Experience with cloud technologies (preferably AWS or GCP) and S3.
• Experience with data lake technologies like Iceberg.
• Bachelor’s degree in computer science or a related field (or equivalent experience).

Preferred Skills

• Experience with data modelling and schema design.
• Experience with DevOps practices and CI/CD pipelines.

Key Skills

Education

Bachelor's degree