Description

You will design and build custom AWS utilities that enable data engineers and analysts to query disparate data sources, specifically bridging the gap between S3-based tables and Amazon Redshift.

This role is on-site.

Responsibilities

• Design and develop resilient, enterprise-grade AWS utilities for seamless cross-platform data querying.
• Build and maintain scalable data pipelines using AWS Glue, PySpark, and Spark SQL to ingest and transform data.
• Implement automated recovery mechanisms and ensure solution reliability in multi-technology environments.
• Utilize CloudFormation to manage and deploy infrastructure components.
• Create tools to streamline analyst workflows, including automated processes for converting SQL outputs.

Required Skills

• 6+ years of Data Engineering or IT experience.
• 4+ years of AWS cloud development experience.
• Expert-level proficiency in PySpark and Spark SQL.
• Deep knowledge of AWS core services: Glue, S3, Lambda, IAM, and Step Functions.
• Strong SQL skills with experience in Athena and Presto.
• Experience with Dremio implementation (reflections, views, system tables).
• Proficient in Python programming.
• Hands-on experience with Git and CI/CD pipelines.
• Bachelor's degree in Computer Science, Engineering, Information Systems, or related discipline.

Preferred Skills

• Familiarity with Tableau for end-user data consumption.

Key Skills

Education

Any Graduate