Description

Design and maintain scalable data pipelines and back-end systems using Spark, Python, Scala, and Java.

This role is on-site.

Responsibilities

• Build and optimize ETL processes with efficient SQL queries and DataFrames.
• Develop real-time streaming pipelines using MSK/Kafka.
• Implement data storage solutions on AWS or GCP, leveraging S3 and Iceberg.
• Ensure data quality, integrity, and security across the platform.
• Troubleshoot pipeline issues and collaborate with data scientists on requirements.

Required Skills

• 5+ years of back-end development with a focus on data engineering.
• Strong proficiency in Spark, Python, Scala, and Java.
• Expertise in SQL and relational databases.
• Experience with cloud technologies (AWS or GCP).
• Hands-on experience with MSK/Kafka for streaming.
• Familiarity with S3 and data lake technologies like Iceberg.

Preferred Skills

• Experience with data modeling and schema design.
• Knowledge of DevOps practices and CI/CD pipelines.

Key Skills

Education

Any Graduate