Description

You will design and implement data pipelines migrating data from Data Lake to Lakehouse architecture.

Responsibilities

• Develop and optimize Spark-based ETL/ELT pipelines using Spark Refiner or similar frameworks.
• Build scalable data processing workflows leveraging AWS services like S3, Glue, EMR, Lambda, and Step Functions.
• Integrate and manage data ingestion into Snowflake for analytics consumption.
• Perform data modelling for Lakehouse layers (Bronze, Silver, Gold) and ensure data quality and lineage.
• Implement CI/CD pipelines for data engineering workflows and support data validation.

Required Skills

• 12+ years of professional experience in data engineering.
• Strong experience with Apache Spark (PySpark / Scala).
• Hands-on expertise with AWS Data Ecosystem: S3, Glue, EMR, Lambda, Step Functions, IAM.
• Experience building large-scale ETL/ELT pipelines.
• Strong knowledge of Snowflake, including Snowpipe Data loading.
• Excellent SQL and data modelling skills.
• Understanding of Data Lake and Lakehouse architectures.
• Experience with data migration strategies and validation techniques.
• Familiarity with DevOps and CI/CD practices.

Preferred Skills

• Experience with Delta Lake, Iceberg, or Hudi.
• Familiarity with Airflow or other workflow orchestration tools.
• Exposure to streaming platforms such as Kafka or Kinesis.

Key Skills

Education

Any Gradute