Description

Key Responsibilities

Build, implement, and optimize enterprise ETL/ELT pipelines using Databricks, Python, PySpark, and AWS Glue.

Migrate existing jobs from Amazon EMR to Databricks.

Extract and integrate data from legacy and modern sources.

Support large-scale, distributed data processing and cloud data platforms.

Ensure data quality, availability, and reliability across financial and reporting systems.

Provide production support and troubleshoot pipeline issues.

Implement CI/CD pipelines using Git and GitHub Actions.

Create technical documentation and job aids, and train analysts.

Collaborate with business analysts and stakeholders in an Agile/Scrum environment.

Required Qualifications

Bachelor’s degree in Computer Science, Engineering, Information Systems, or a related field.

7+ years of enterprise Data Engineering experience.

Strong Python development and advanced SQL programming skills.

Extensive hands-on experience with Apache Spark, PySpark, Databricks, and AWS Glue.

Experience with AWS S3, Athena, Lambda, Step Functions, and EventBridge.

Proven experience building enterprise ETL/ELT pipelines.

Strong understanding of distributed data processing, data lakes, and Medallion Architecture.

Strong knowledge of dimensional data modeling.

Experience with Git, GitHub Actions, and CI/CD implementation.

Familiarity with Agile/Scrum methodologies.

Excellent communication and stakeholder management skills.

Additional Technical Stack

Infrastructure as Code: CloudFormation / Terraform

AI/ML Platform: Amazon SageMaker

Preferred Qualifications

Financial Services or Asset Management experience.

Knowledge of Apache Iceberg, Delta Lake, or Hudi.

Experience with Apache Airflow.

Knowledge of Kafka or Amazon Kinesis streaming.

Experience supporting AI/ML and advanced analytics platforms.

AWS Professional or Specialty certifications

Key Skills

Education

Bachelor's degree