Description

Responsibilities

• Design and implement high-performance, scalable ETL and ELT pipelines using Databricks, Apache Spark, and AWS data services
• Orchestrate data extraction and transformation from legacy and modern data sources
• Migrate data processing jobs from EMR to Databricks
• Develop and maintain CI/CD pipelines for data infrastructure
• Collaborate with business analysts and stakeholders to ensure data availability, quality, and integrity
• Provide expert-level production support and troubleshooting for data platforms
• Create documentation and job aids to support team operations and training
• Implement dimensional data modeling and data lake architectures (Medallion Architecture)

Qualifications

• Required
  • Bachelor's degree in Computer Science, Engineering, Information Systems, or related discipline
  • 8+ years of enterprise Data Engineering experience
  • Strong hands-on Python development experience
  • Advanced SQL programming skills
  • Extensive experience with Apache Spark and PySpark
  • Strong hands-on experience with Databricks
  • Hands-on experience with AWS data services: S3, AWS Glue, Athena, Lambda, Step Functions, and EventBridge
  • Experience building enterprise ETL and ELT pipelines
  • Strong understanding of distributed data processing
  • Experience working with large-scale cloud data platforms
  • Strong knowledge of dimensional data modeling
  • Experience implementing CI/CD pipelines
  • Experience with Git and GitHub Actions
  • Familiarity with Agile/Scrum methodologies
  • Excellent communication and stakeholder management skills
• Preferred
  • Financial Services or Asset Management industry experience
  • Knowledge of Apache Iceberg, Delta Lake, or Hudi
  • Experience with Apache Airflow
  • Knowledge of Kafka or Kinesis streaming
  • Experience supporting AI/ML and advanced analytics platforms
  • AWS Professional or Specialty Certifications

Key Skills

Education

Bachelor's degree