Description

You will design, build, and optimize scalable data pipelines and cloud-native data platforms using PySpark and Snowflake.

This role is on-site.

Responsibilities

• Build production-grade PySpark pipelines for large-scale batch and streaming data processing, including complex transformations and deduplication logic.
• Develop and maintain Snowflake data models, warehouses, Iceberg tables, Dynamic Tables, and Materialized Views for analytics.
• Optimize PySpark job performance through partitioning, caching, broadcast joins, and shuffle optimization.
• Tune Snowflake workloads via clustering, micro-partition pruning, query profiling, and warehouse sizing.
• Migrate legacy on-prem ETL workloads to cloud-based PySpark and Snowflake solutions while ensuring data quality and governance.

Required Skills

• 8+ years of hands-on data engineering experience in enterprise environments.
• Strong expertise in PySpark, including reading S3 Parquet/Delta files and handling large datasets.
• Proven experience in PySpark performance tuning for very large datasets.
• Deep hands-on experience with Snowflake: SnowSQL, Snowpipe, Streams, Tasks, Iceberg Tables, and Dynamic Tables.
• Strong Snowflake performance tuning skills (clustering, caching, cost optimization).
• Proven AWS experience with S3, Glue, EMR, Lambda, Step Functions, IAM, and Redshift.
• Advanced SQL skills including complex joins, window functions, CTEs, and performance tuning.
• Strong Python programming skills for utilities, automation, and orchestration.
• Experience with orchestration tools such as Airflow or AWS Step Functions.

Preferred Skills

• Experience modernizing legacy ETL (DataStage, Informatica) to cloud-native architectures.
• Solid understanding of data warehousing, ELT/ETL patterns, and lakehouse architectures.

Key Skills

Education

Bachelor's degree