Description

Own the design, development, and maintenance of scalable data lakes and ETL pipelines using AWS and PySpark.

Responsibilities

• Build and optimize ETL pipelines using PySpark and advanced SQL on AWS Glue.
• Manage data storage and processing workflows within AWS S3 and Glue.
• Orchestrate complex data workflows using AWS Lambda and Step Functions.
• Develop and maintain data lake architectures to support analytics and reporting.
• Collaborate with stakeholders to define data requirements and ensure data quality.

Required Skills

• 10+ years of professional experience in data engineering.
• Expert-level proficiency in PySpark for large-scale data processing.
• Advanced SQL skills for complex data manipulation and querying.
• Hands-on experience with AWS services: Glue, S3, Lambda, and Step Functions.
• Proven track record of building and maintaining ETL pipelines.
• Experience working with data lakes and distributed data storage.
• Bachelor's degree in Computer Science, Engineering, or a related field.

Preferred Skills

• Experience with Databricks is a plus.

Key Skills

Education

Any Gradute