Description

You will design, develop, and optimize batch ETL pipelines within a cloud-based Lakehouse architecture.

Responsibilities

• Build and implement batch ETL pipelines using Databricks, PySpark, and Spark SQL.
• Ensure data quality and integrity through validation, cleansing, and enrichment using Delta Lake ACID features.
• Optimize Spark job performance, partitioning, and dependencies to meet processing SLAs.
• Enforce data security, encryption, and lineage to maintain compliance with banking regulations.
• Automate pipeline workflows using Databricks Jobs, Delta Live Tables, and Airflow.

Required Skills

• 3+ years of hands-on experience in data engineering with Databricks and Apache Spark.
• Proficiency in PySpark and Spark SQL.
• Strong command of Python and SQL.
• Experience with Delta Lake and Lakehouse architecture.
• Hands-on experience with Azure, AWS, or GCP cloud platforms.
• Knowledge of cloud data services including Azure Data Lake Storage, Azure Data Factory, S3, or BigQuery.
• Experience with pipeline orchestration using Airflow, Databricks Jobs, or ADF.
• Familiarity with version control (Git) and CI/CD for data pipeline deployment.
• Bachelor’s degree in Computer Science or a related field.

Preferred Skills

• Experience with banking or financial data, including transactions or risk analytics.
• Knowledge of real-time streaming using Kafka or Spark Structured Streaming.
• Familiarity with DataOps and MLOps practices.

Key Skills

Education

Bachelor's degree