Description

You will build and maintain data pipelines using Spark technologies.

Responsibilities

• Develop batch and streaming data processing jobs using Apache Spark.
• Implement data transformations using SQL frameworks.
• Orchestrate and manage data workflows using Airflow or Control-M.
• Integrate data sources via Kafka or Solace and manage data in object stores like S3.

Required Skills

• 5+ years of professional experience in data engineering.
• Strong expertise in Apache Spark (batch and streaming).
• Proficiency in Python, Scala, or Java.
• Experience with SQL transformation frameworks, preferably DBT.
• Hands-on experience with data Lakehouse formats (Iceberg, Delta Lake, Hudi).
• Familiarity with Kafka and Solace message queuing systems.
• Knowledge of orchestration tools, specifically Airflow or Control-M.
• Experience working with object stores (S3, MinIO).

Key Skills

Education

Any Graduate