Description

You will build and maintain data pipelines using distributed computing frameworks and cloud technologies.

Responsibilities

• Develop data processing workflows using Spark, PySpark, and Hive.
• Manage data streaming and ingestion via Kafka.
• Implement CI/CD pipelines to automate deployment processes.
• Write technical documentation and communicate architectural decisions to peers.
• Collaborate with team members to meet project productivity metrics and timelines.

Required Skills

• 5 to 8 years of experience in big data engineering.
• Proficiency in Python and PySpark.
• Hands-on experience with Apache Spark and Hive.
• Experience with GCP and BigQuery.
• Working knowledge of Kafka for real-time data streaming.
• Experience with MySQL and NoSQL databases.
• Familiarity with Hadoop ecosystems.
• Practical understanding of CI/CD workflows.

Preferred Skills

• 2 years tenure in previous role.

Key Skills

Education

Not specified