You will build and maintain backend systems focusing on data engineering tasks.
Responsibilities
- Design, develop, and maintain data pipelines using Spark, Python, Scala, and Java.
- Write optimized SQL queries for ETL processes.
- Manipulate and analyze large datasets using DataFrame operations.
- Implement data storage and processing solutions leveraging AWS or GCP.
- Build and maintain real-time data streaming pipelines with MSK/Kafka.
- Ensure data quality, integrity, and security across pipelines.
- Troubleshoot and resolve issues in data pipelines and back-end systems.
Required Skills
- 5+ years of experience in back-end development with a focus on data engineering.
- Strong proficiency in Spark, Python, Scala, and Java.
- Expertise in SQL and working with relational databases.
- Experience with cloud technologies, specifically AWS or GCP.
- Experience with message streaming platforms like MSK/Kafka.
- Experience with S3 or similar object storage.
- Experience with data lake technologies like Iceberg.
- Solid understanding of data warehousing concepts.
- Excellent communication and collaboration skills.