Description

You will design and implement large-scale data engineering ETL processes and manage migrations to Spark environments.

This role is on-site.

Responsibilities

• Develop Spark applications for batch and streaming workloads using PySpark, Scala, Java, and Python.
• Implement ETL pipelines and execute data migrations to Spark architectures.
• Manage data workflows using Airflow or similar orchestration engines.
• Build and maintain streaming architectures using Kafka and Spark Structured Streaming.

Required Skills

• 5+ years of experience in data engineering.
• Proficiency in Spark development with PySpark.
• Strong programming skills in Scala, Java, and Python.
• Hands-on experience with Kafka and Spark Structured Streaming or Dstream.
• Experience implementing ETL processes and migrations.
• Proficiency with Jupyter notebooks or equivalent developer tools.
• Experience with Airflow or other workflow orchestration engines.

Preferred Skills

• Experience with Flink, Kudu, or NiFi for streaming and transformation.
• Knowledge of CI/CD and workflow automation.
• Experience with Informatica workflow migration.

Key Skills

Education

Any Graduate