Description

Job Summary

We are seeking a talented Senior Data Engineer to join our infrastructure support team. This role is focused on designing and developing robust AWS utilities that enable our data engineers and analysts to seamlessly query across multiple data sources, specifically bridging the gap between S3-based tables and Amazon Redshift. As we transition to a multi-technology data environment, you will be responsible for building resilient, enterprise-grade solutions that prioritize reliability and quick recovery.

Key Responsibilities

• Design & Development: Lead the design and development of custom AWS utilities that allow analysts to query disparate data sources (S3 and Redshift) with minimal effort and high productivity.
• Pipeline Engineering: Build and maintain scalable data pipelines using AWS Glue, PySpark, and Spark SQL to ingest and transform data across platforms.
• Resiliency & Scalability: Apply an "enterprise-grade " development mindset. Build solutions that are resilient, scalable, and capable of quick automated recovery in the event of failure.
• Utility Creation: Create tools to streamline the user experience for analysts, such as building automated processes to convert SQL outputs for multi-source tables.
• Infrastructure Ops: Utilize CloudFormation to manage and deploy infrastructure components.

Required Technical Skills

• AWS Core: Deep proficiency in AWS Glue, S3, Lambda, IAM, and Step Functions.
• Data Engineering: Expert-level knowledge of PySpark and Spark SQL.
• Querying & SQL: Extensive experience with Athena and Presto; strong SQL skills for complex data manipulation.
• Infrastructure as Code: Proven experience with CloudFormation for operational excellence.
• Development Mindset: Ability to write clean, maintainable code for automation and custom utility development.
• Dremio: Experience with Dremio implementation (reflections, views, using system tables).

Required Qualifications
 

• 6-8+ years of Data Engineering or IT experience.
• 4+ years of AWS cloud development experience.
• Expert Python programming skills.
• Advanced SQL development experience.
• Strong hands-on experience with Apache Spark / PySpark.
• Experience building cloud-native ETL pipelines.
• Strong understanding of distributed computing.
• Experience designing scalable data models.
• Experience with Git and CI/CD pipelines.
• Strong analytical and problem-solving skills.
• Build scalable and auto recoverable pipelines.
• Bachelor's degree in Computer Science, Engineering, Information Systems, or related discipline.

Nice to Have

• Tableau: Familiarity with Tableau for end-user data consumption

Key Skills

Education

Any Graduate