Description

Key Responsibilities

• Design & Development: Lead the design and development of custom AWS utilities that allow analysts to query disparate data sources (S3 and Redshift) with minimal effort and high productivity.
• Pipeline Engineering: Build and maintain scalable data pipelines using AWS Glue, PySpark, and Spark SQL to ingest and transform data across platforms.
• Resiliency & Scalability: Apply an "enterprise-grade" development mindset. Build solutions that are resilient, scalable, and capable of quick automated recovery in the event of failure.
• Utility Creation: Create tools to streamline the user experience for analysts, such as building automated processes to convert SQL outputs for multi-source tables.
• Infrastructure Ops: Utilize CloudFormation to manage and deploy infrastructure components.

Required Technical Skills

• AWS Core: Deep proficiency in AWS Glue, S3, Lambda, IAM, and Step Functions.
• Data Engineering: Expert-level knowledge of PySpark and Spark SQL.
• Querying & SQL: Extensive experience with Athena and Presto; strong SQL skills for complex data manipulation.
• Infrastructure as Code: Proven experience with CloudFormation for operational excellence.
• Development Mindset: Ability to write clean, maintainable code for automation and custom utility development.
• Dremio: Experience with Dremio implementation (reflections, views, using system tables).

Required Qualifications

• 6-8+ years of Data Engineering or IT experience.
• 4+ years of AWS cloud development experience.
• Expert Python programming skills.
• Advanced SQL development experience.
• Strong hands-on experience with Apache Spark / PySpark.
• Experience building cloud-native ETL pipelines.
• Strong understanding of distributed computing.
• Experience designing scalable data models.
• Experience with Git and CI/CD pipelines.
• Strong analytical and problem-solving skills.
• Build scalable and auto recoverable pipelines.
• Bachelor's degree in Computer Science, Engineering, Information Systems, or related discipline

Key Skills

Education

Bachelor's degree