Description

Design, build, and maintain high-performance ETL/ELT data pipelines using Python and PySpark for distributed computing across large-scale datasets.

This role is on-site.

Responsibilities

• Develop and manage data processing jobs using PySpark, taking full ownership of the data workflow from ingestion to validation.
• Write and optimize complex, performant SQL queries for data extraction, integrity checks, and performance tuning.
• Implement DevOps practices for automated deployment and orchestration of Python applications using Docker, Jenkins, and Terraform.
• Explore and increase adoption of AI/ML tools like Copilot and Claude to accelerate development and build models for data gaps.
• Collaborate with data architects to propose solutions and maintain expertise on data pipelines and systems.

Required Skills

• Expert-level proficiency in Python, including Pandas and NumPy.
• Solid hands-on experience with PySpark for building scalable data workflows.
• Expert-level knowledge of complex SQL queries (Oracle or Snowflake) with proven performance tuning skills.
• Robust experience with AWS cloud services: Glue, S3, Lambda, Redshift, DynamoDB, Athena, ECS, EventBridge, OpenSearch, RDS.
• Proficiency in ETL/ELT methodologies, data quality, validation, and anomaly detection.
• Working experience with Unix scripting and automation.
• Bachelor's degree and 5+ years of experience.

Preferred Skills

• Familiarity with AI/ML and Large Language Model (LLM) approaches to data analysis.
• Experience with BI Reporting tools such as Power BI or Tableau.
• Prior experience in the investment data domain.

Key Skills

Education

Bachelor's degree