Description

Key Responsibilities

• Design, develop, and maintain scalable data engineering solutions using Databricks, PySpark, Spark SQL, and Apache Spark
• Build and optimize end-to-end ETL/ELT pipelines integrating Databricks with AWS S3, Snowflake, Apache Airflow, and AWS Glue
• Process and transform large-scale structured and semi-structured datasets for enterprise reporting, analytics, and regulatory requirements
• Develop reusable frameworks for data ingestion, cleansing, transformation, validation, reconciliation, enrichment, and business rule implementation
• Implement data quality checks, monitoring, and governance controls
• Perform performance tuning and optimization of Spark applications and Databricks workloads
• Collaborate with stakeholders, architects, and analytics teams to deliver scalable solutions
• Support advanced analytics initiatives including customer segmentation, campaign analytics, AML/regulatory reporting, and data validation

Required Skills

• Databricks and Lakehouse Architecture
• PySpark, Spark SQL, Apache Spark
• AWS S3, AWS Glue
• Snowflake
• Apache Airflow
• Data modeling, data warehousing, and big data processing
• Structured and semi-structured data formats: JSON, Parquet, Avro, CSV
• Data quality frameworks and monitoring solutions
• Strong SQL and data analysis skills

Preferred Skills

• Experience in AML, regulatory reporting, compliance, or financial data domains
• CI/CD and DevOps practices for data engineering
• Delta Lake, Databricks Workflows, Unity Catalog
• Agile/Scrum delivery methodologies

Key Skills

Education

Bachelor's degree