Description

You will build and maintain large-scale data processing pipelines for complex data transformations.

Responsibilities

• Develop data processing logic using PySpark and Spark Scala.
• Manage and process JSON data streams through Kafka.
• Orchestrate data workflows using Airflow.
• Write and optimize complex SQL and Hive queries for data transformation.
• Implement CI/CD processes within the data engineering lifecycle.

Required Skills

• 5+ years of overall experience with 3+ years in relevant big data roles.
• Proficiency in Python programming.
• Extensive experience with PySpark and Spark Scala.
• Hands-on expertise with Kafka and JSON data handling.
• Strong knowledge of SQL and Hive.
• Experience with scheduling tools like Airflow.
• Understanding of CI/CD workflows.

Preferred Skills

• Experience with Google Cloud Platform (GCP).
• Familiarity with data visualization tools like PowerBI or Superset.
• Knowledge of AWS or Azure data services.

Key Skills

Education

Bachelor's degree in Computer Science