Description

Design, develop, and maintain scalable data pipelines and back-end systems for large datasets.

This role is on-site.

Responsibilities

• Design and maintain data pipelines using Spark, Python, Scala, and Java.
• Write optimized SQL queries for ETL processes.
• Implement real-time data streaming pipelines using MSK/Kafka.
• Build data storage and processing solutions on AWS or GCP.
• Troubleshoot pipeline issues and ensure data quality and security.

Required Skills

• 5+ years of back-end development and data engineering experience.
• Strong proficiency in Spark, Python, Scala, and Java.
• Expertise in SQL and relational databases.
• Experience with DataFrames for large-scale data manipulation.
• Hands-on experience with AWS or GCP cloud technologies.
• Experience with MSK/Kafka for message streaming.
• Experience with S3 or similar object storage.
• Knowledge of data lake technologies like Iceberg.

Preferred Skills

• Experience with data modelling and schema design.
• Familiarity with DevOps practices and CI/CD pipelines.

Key Skills

Education

Any Graduate