Description

You design, build, and maintain scalable data pipelines and platforms to support enterprise analytics, reporting, and AI/ML initiatives.

This role is on-site.

Responsibilities

• Design and develop scalable data pipelines for batch and real-time processing using Python, SQL, and Spark.
• Build and optimize ETL/ELT workflows and data ingestion processes from structured and unstructured sources.
• Implement scalable data models and manage data warehousing using Azure Synapse, Snowflake, or Redshift.
• Develop automated testing, monitoring, and deployment processes for data pipelines using CI/CD and Git.
• Troubleshoot production issues, optimize performance and cost efficiency, and ensure data quality and governance.

Required Skills

• 10+ years of experience in Data Engineering with strong programming skills in Python and SQL.
• Hands-on experience with Apache Spark/PySpark and Databricks for large-scale data processing.
• Proficiency in ETL/ELT design patterns and data modeling concepts.
• Experience with Azure (ADF, ADLS) or AWS (Glue, S3) cloud-based data engineering solutions.
• Knowledge of relational databases, data warehousing, and version control (Git).
• Experience implementing CI/CD pipelines and automated testing for data infrastructure.

Preferred Skills

• Experience with Docker containerization for data engineering workflows.
• Background in Agile teams within the financial services domain.

Key Skills

Education

Bachelor's degree