Description

You will own the design, development, and maintenance of data pipelines.

Responsibilities

• Design, develop, and maintain data pipelines using Spark, Python, Scala, and Java.
• Write efficient and optimized SQL queries for ETL processes.
• Manipulate and analyze large datasets using DataFrame operations.
• Implement data storage and processing solutions using AWS or GCP.
• Build and maintain real-time data streaming pipelines using MSK/Kafka.
• Ensure data quality, integrity, and security across all systems.
• Collaborate with data scientists to define data requirements and deliver solutions.
• Troubleshoot and resolve issues within data pipelines and back-end systems.

Required Skills

• 5+ years of experience in back-end development focused on data engineering.
• Strong proficiency in Java, Spark, Python, and Scala.
• Expertise in SQL and working with relational databases.
• Experience with AWS or GCP cloud technologies.
• Experience with message streaming platforms like Kafka.
• Experience utilizing S3 or similar object storage.
• Experience with data lake technologies such as Iceberg.
• Solid understanding of data warehousing concepts.
• Bachelor's degree in Computer Science or a related field.

Key Skills

Education

Bachelor's degree