Description

• Design and build scalable, production-grade data pipelines using PySpark for batch and streaming processing, including data transformations, deduplication, and performance optimization (partitioning, caching, shuffle optimization)
• Develop cloud-native data solutions on AWS, leveraging S3, Glue, EMR, Lambda, Step Functions, and Redshift
• Engineer comprehensive Snowflake data platforms including warehouses, schemas, data models, Iceberg Tables, Dynamic Tables, and Materialized Views
• Modernize legacy ETL workloads to cloud-based ELT/ETL patterns and implement orchestration using Airflow or AWS Step Functions
• Optimize Snowflake and PySpark workloads through tuning strategies and performance monitoring to ensure efficiency at scale
• Ensure data quality, governance, and documentation across all data pipelines, including technical designs and runbooks
• Collaborate with data architects, analysts, and stakeholders to align solutions with business requirements

Requirements

• 8-10 years of experience as a Data Engineer with proven expertise in PySpark, Snowflake, and AWS cloud services
• Advanced proficiency in Python and SQL with the ability to write optimized, production-grade code
• Strong experience with AWS services including S3, Glue, EMR, Lambda, Step Functions, and Redshift
• Demonstrated expertise in Snowflake including Iceberg Tables, Dynamic Tables, and Materialized Views
• Solid understanding of data pipeline design, ETL/ELT patterns, data modeling, and data warehousing best practices
• Experience with orchestration tools such as Airflow or AWS Step Functions
• Strong communication skills and ability to work collaboratively with cross-functional technical and business teams

Key Skills

Education

Bachelor's degree