Description

Build and maintain scalable, resilient data pipelines and custom AWS utilities for analysts.

This role is on-site.

Responsibilities

• Design and develop custom AWS utilities to streamline analyst queries across S3 and Redshift.
• Build and maintain scalable data pipelines using AWS Glue, PySpark, and Spark SQL.
• Implement enterprise-grade resiliency with automated recovery mechanisms for distributed systems.
• Create tools to automate SQL output conversion and streamline user experiences.
• Manage and deploy infrastructure components using CloudFormation.

Required Skills

• 6-8+ years of Data Engineering or IT experience.
• 4+ years of hands-on AWS cloud development.
• Expert-level Python programming and PySpark/Spark SQL.
• Deep proficiency in AWS Glue, S3, Lambda, IAM, and Step Functions.
• Extensive experience with Athena and Presto for complex data manipulation.
• Proven experience with CloudFormation for Infrastructure as Code.
• Experience with Dremio implementation (reflections, views, system tables).
• Strong understanding of distributed computing and scalable data models.

Preferred Skills

• Experience with Tableau for data visualization.
• Proficiency with Git and CI/CD pipelines.

Key Skills

Education

Bachelor's degree