Description

You will design and implement data engineering ETL processes and manage large-scale migrations to Spark.

Responsibilities

• Develop Spark applications for both batch and streaming workloads.
• Implement ETL pipelines and execute data migrations to Spark environments.
• Manage data workflows using Airflow or similar orchestration engines.
• Build and maintain streaming architectures using Kafka and Spark Structured Streaming.

Required Skills

• 5+ years of experience in data engineering.
• Proficiency in Spark development with PySpark.
• Strong programming skills in Scala, Java, and Python.
• Hands-on experience with Kafka and Spark Structured Streaming or Dstream.
• Experience implementing ETL processes and migrations.
• Proficiency with Jupyter notebooks or equivalent developer tools.
• Experience with Airflow or other workflow orchestration engines.
• Any Graduate degree.

Preferred Skills

• Experience with Flink, Kudu, or NiFi for streaming and transformation.
• Knowledge of CI/CD and workflow automation.
• Experience with Informatica workflow migration.

Key Skills

Education

Any Graduate